From the DCY2K List server:

Hi everyone,

This just came across on the EUY2k forum... thought all might like to see it.

I would love to hear analysis from those who are more technically-minded.

Bob

=====================================================================

I got this from John Koskinen today in response to an email. I will leave its analysis to those brighter than I.

(To the extent it says in part, that problems exist in some embedded system even though dates aren't expressly used, it undermines the suggestion that untested embedded systems in satellites are ok "because they don't use dates".)

Memorandum from John A. Koskinen

Subject: Summary of Discussion with Embedded Systems Technicians

Attached is a summary of a recent discussion that I organized among technical experts that have been working on testing and fixing embedded systems across the manufacturing, electric power, oil, gas, telecommunications, shipping, bio-medical device, and defense industries.

It outlines the discussion at the meeting and makes a number of generally agreed to statements concerning the types of embedded systems that have Y2K risk, when embedded systems will fail, difficulties in testing for Y2K problems in embedded systems, and risks of not fixing the problem in advance.

Those at the meeting reported that the organizations they are working with have addressed all these facets of testing and fixing the problem, but that they remain vigilant about changing manufacturers statements of compliance.

I believe that the discussion and agreed to statements are important for all those working on the embedded system problem to hear. Therefore, I am forwarding this summary to you and the other working groups. Feel free to disseminate it further.

Attachment

PRESIDENTS COUNCIL ON YEAR 2000 CONVERSIONS

MEETING ON Y2K EMBEDDED SYSTEMS
Tuesday, November 9, 1999
American Society of Association Executives Building
1575 I Street, Washington, DC

Participants in the meeting included technicians that had done work in the bio-medical, defense, electric power, gas, manufacturing, oil, shipping, and telecommunications industries. To help with the discussion, an agenda was provided with discussion statements concerning the types of embedded systems potentially atY2K risk, difficulties in testing for such embedded systems and fixes for problems found. Those statements were revised during the meeting and the agreed upon final statements are presented below, along with a brief summary of the discussion that led to the final statement.

Types of embedded systems found to have a Y2K risk:

Final Statement: Embedded systems are at risk of problems during Y2K rollover if they conduct a calculation that depends on a representation of the date. The date could be in relative or absolute form.

The participants presented a number of specific cases where they had found Y2K problems in embedded systems. Several of these involve calculations of time increments inside an embedded system without the date being displayed or apparently used. In these instances an embedded system calculates the time interval by subtracting seconds from seconds, minutes from minutes, hours from hours, and calendar dates from calendar dates.

All except one of the examples were large, complex processes where embedded systems inter-relate with each other and, in some cases, with external computer systems. The one example was of a stand-alone embedded system that was unconnected to others that did not apparently involve dates.

That example lead to a discussion about the need for a continuous power source being available for any such devices to function, and it was pointed out that in some sectors there are many such devices, but that few problems had been found in them.

There was considerable discussion of potential failure rates of embedded systems. Estimates ranged from a 1 - 2% potential failure rate of processes containing embedded systems in some sectors to 4 - 6% in others, but no conclusion was reached. An important distinction was made between failure of an embedded system, which may not cause a process or device to fail in operation, and failure of a process or device due to an embedded system. The former represents the estimates above, and the latter is much less prevalent.

The remainder of the discussion during the meeting focussed on large, complex processes that contain embedded systems. The question of having a real time clock or access to a clock was discussed and examples were presented where the time was set by a process controller and transmitted to other embedded processors involved in the process. Other examples of problems were discussed where time was used apparently to calculate relative increments (e.g. day of the week) as opposed to absolute dates.

When embedded systems will fail:

Final Statement: Where possible, all mission critical systems should be tested end-to-end, whether or not the systems appear to have date sensitive functions. Failure to do so means a small level of risk has been assumed that, at minimum, should be addressed with a contingency plan.

The discussion that lead to this statement began with a presumption that embedded systems involved in calculating time increments, as well as those that apparently computed dates, are at Y2K risk.

During the discussion the statement to test mission critical systems whether they have a date function or not was almost agreed to, until it was pointed out one can only test those types of devices with end-to-end testing.

This statement was focussed on mission critical systems because it is difficult and expensive to conduct such testing. The term mission critical systems was used to include safety critical systems as well as other systems where the cost of failure would be high. Therefore, while the statement says the risk of failure is low, the impact of any such failure would be high. The statement also recommends a contingency plan to help mitigate risk -- such a plan should not be viewed as an alternative to testing because detection of a failure may be difficult and a failure could cause substantial collateral damage before it is detected.

Final Statement: The majority of failures of embedded systems are expected to occur on or about December 31st through January 1st. However, simply turning a system off during that time frame is generally not a solution.

The discussion explored the question of whether the time of primary risk of failure was during the rollover time. It was generally agreed that the vast majority of failures in embedded systems are likely to occur over that period.

On the specific question of whether Greenwich Mean Time would be a time of high failure, it was stated that most failures would likely occur at 12:00 local time, although some would also occur on Greenwich time. During the discussion, there was a concern raised that the statement may lead to the ineffective solution of turning off systems during the rollover period. Therefore, the specific admonition not to rely on that work-around was included in the statement.

Final Statement: One can have two apparently identical systems of which one will not have a Y2K problem but the other will have operating difficulties. However, the chances of this are small.

The likelihood of failure of one of two identical systems, as described in this statement was considered to be very small, but, again, it was agreed that all mission critical systems needed to be tested.

Difficulties in testing for embedded systems at risk:

Final Statement: Organizations that have relied on a device manufacturers declaration of Y2K compliance are at risk if they do not keep up with the most recent manufacturers statements.

The discussion concerned cases where testing had brought into question manufacturers statements of the readiness of their products. A number of instances were cited where problems had been found both externally by users that had tested and by manufacturers themselves. While the changes needed to remedy such problems have normally been made quickly available, the concern was expressed that many organizations were not aware of or taking advantage of those fixes.

Final Statement: Some interconnection problems among embedded systems can only be revealed by end-to-end testing.

The discussion concerned how to test for problems in embedded systems. There was considerable discussion of difficulties of testing in operational environments and the risks and complexities of end-to-end testing. However, a number of examples were cited to show that one could not find all potential problems in complex, interconnected embedded processes without end-to-end testing.

Fixes:

Final Statement: Anyone taking a fix-on-failure approach for Y2K, particularly with embedded systems, runs a significant risk of collateral damage and a difficult recovery.

There was little discussion leading to this statement. Remedying the kinds of Y2K problems participants had found in embedded systems was difficult and time-consuming.

Final Statement: After a full and careful technical assessment, there may be administrative or operational workarounds to many Y2K problems involving embedded systems.

While simply turning a system off during the rollover is not normally an effective administrative work-around, in some instance it could be. Similarly, setting the year back so that Y2K does not occur may be a work-around in some instances. However, before using these or any other ways to work-around the Y2K problem, all agreed that a thorough assessment of the full implications of the work-around was necessary.

Final Statement: Even those that have conducted thorough testing need to develop contingency plans for mission critical processes and exercise them.

There was little discussion of this statement, in light of the earlier statements that indicate the risk of Y2K problems.

____________________________________________________________________

-- Helium (Heliumavid@yahoo.com), December 02, 1999

Answers

Ouch. Was it formatted like that over there?

-- dagnabbit I'm (blind@inbotheyes.now), December 02, 1999.

From Jim Lord's post of today..

(snip)

In addition he indicated his company is finding embedded chip failure rates that are much higher (5-10%) than the optimistic figures being reported by the likes of Bennett/Koskinen (0.3-0.5%). Also, it has been his experience that many utility (and other) companies are not undergoing independent verification and validation (IV&V) of their Y2K projects because they dont want the dirty paper in their files. The implication is that in the event of lawsuits, no IV&V is preferable to a failed IV&V.

(snip)

It's all coming together now and the picture is bleak. After reading about embeddeds for two years, I have concluded that that testing has not been done on the majority because of the time and expense involved which corporations underestimated from the beginning. Updates in compliance from manufacturers can't be undertaken because of the time factor now. Does any one now doubt that at least one disaster of epic proportions is emminent at this point?

-- PJC (paulchri@msn.com), December 02, 1999.

Nearly five thousand U.S. chemical facilities are storing greater quantities of extremely hazardous substances than were released in Bhopal chemical accident, according to the report, "Accidents Waiting to Happen: Hazardous Chemical Storage in the U.S."

LINK

-- PJC (paulchri@msn.com), December 02, 1999.

Helium, Just a quick "credit" here, the email was received from Koskinen by, and posted to EUY2K by, F. Snyder Gokey. Snyder has posted a number of noteworthy items there, I always look forward to his posts.

-- FactFinder (FactFinder@bzn.com), December 02, 1999.

My eyes glazed over before I got to: 'But don't worry, all will be repaired in three days' -- that was in the text, right?

Did Kosky really send you this? -- seems too honest for a Clinton Administration public relations flak who wouldn't know a computer if he fell over one!

-- Richard Greene (rgreene2@ford.com), December 02, 1999.

Well that was a challanging read. Almost feel like cleaning it up.

Fine time to get into this. Just another indication that some folks can't tell time.

""Fixes: Final Statement: Anyone taking a fix-on-failure approach for Y2K, particularly with embedded systems, runs a significant risk of collateral damage and a difficult recovery. There was little discussion leading to this statement. Remedying the kinds of Y2K problems participants had found in embedded systems was difficult and time- consuming.""

-- Brian (imager@home.com), December 02, 1999.

Has anyone shown this to Oprah OR Clinton?

-- GoldReal (GoldReal@aol.com), December 02, 1999.

$#!+ !!! So this must be part of what Dr. Gordon has been referring to. What this Forum has always known, and Koffinsky is just getting a taste of.

We'd just like to say a big F**K YOU to the disgusting troll The Engineer who has been bragging about "minimal type testing" at BPA, and whose snide jerk attitude has endangered the lives of millions of Cascadians.

Engineer, may you be the next fried squirrel.

Guess what, folks? We will see a panic at the end of the year, and it will be the .gov .mil .system that is panicking.

The weeples will panic afterwards.

Prepare for a long visit back to 1900.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), December 02, 1999.

Sysops

Cool! reformated!!

Thanks :o)

-- Brian (imager@home.com), December 02, 1999.

This shit sucks. Where are all the big brained pollies on this one? Credible information from the pres. spinmeister. WTF people?

Can ANYONE with embedded experience please refute this. Tell me it ain't so. Tell me all the pipes and refineries aren't going to go bang? I'm not kidding here people. I have heard a hundred assholes in the oil industry tell me how they were so glad they "didn't find nearly as many problem chips as they thought they would" and how the "vendors supplied them with good info" and "the failure rates were much lower than they thought" blah blah blah.

We're really gonna have a bad shitstorm here shortly if this stuff is true. Please, anyone with good info please post here. We need to hear from you ASAP. Factfinder, Y2K pro, Hamasaki, anyone with some damn knowledge of this stuff out there?

Anyone know anything about Century or Nist? How big, credible etc?

-- Gordon (g_gecko_69@hotmail.com), December 02, 1999.

Oh boy. Ya Know, I have perused through so much polly/troll dung in the last couple of months to last me a life time...but I would sure love to hear some of their bs about now. Or could it be their contracts were up Nov. 30, 1999. I have not had too much time recently, but have only seen Cherri posting. .....Where have all the pollies gone............?

......... test mission critical systems whether they have a date function or not was almost agreed to, until it was pointed out one can only test those types of devices with end-to-end testing.

Heavens no, we can not test end-to-end, for crying out loud, do you know how much time/money/resources that would take. X (crossing- fingers) FOF. ((((BIG SIGH))))

-- anxietyattackrighthererightnow (karlacalif@aol.com), December 02, 1999.

Well if you're waiting on flint don't hold your breath. Two months ago I challenged him toshow how they could have possibly tested all those millions of chips in a typical power plant in less then a year without even shutting the plant down, and all he did was evade the question.

-- Nikoli Krushev (doomsday@y2000.com), December 02, 1999.

Gordon, we're eMailing Dr. Gordon about this thread. See also:

Y2k report says time is running out (...technical report disputes the myth that embedded systems can be ignored if they don't appear to use a date.)

Daley urges testing of embedded systems for Y2k problems (Federal govt on the cutting edge)

-- Ashton & Leska in Cascadia (allaha@earthlink.net), December 02, 1999.

This astounds me..............that this meeting was held just 52 days before the rollover is negligent, criminally negligent. This meeting should have been held in 1997 at the latest.

History in the making.

-- David Waldrip (dwaldrip@aol.com), December 02, 1999.

Gordon et al,

Read it and weep, guys. This posting, along with the NIST/Century and the Mr. CEO info are pretty much in-line with my various oil industry sources (from Calif. to Texas, Louisiana to Pennsylvania-Ohio-Indiana- Illinois oil fields) who've told me essentially the same thing in regards to the oil industry embeddeds problem. Reading these latest threads is (as Yogi Berra would say) "deja vu all over again."

Gordo... I fully concur... where is Factfinder to refute the findings of their hero...Kosky himself... Where is Cherri, Where is "Engineer" where are all the rest of that stinking bunch of morons. You know, judging by the above posts on this thread...its a good thing this is cyberspace or the gang here would be getting a rope about now for a good hangin' party.

-- R.C. (racambab@mailcity.com), December 02, 1999.

God Bless America, My Home Sweet Home



-- the Virginian (1@1.com), December 03, 1999.

this is the result of the conference which is the result of the NIST/Century study.

It's CLEAR to ME that the NIST/Century study was commissioned to validate the prevailing Administration position. Imagine their surprise when it didn't!! NOW they have to go on record as KNOWING that there are GOING TO BE PROBLEMS so they can whimper, from their bunker "We TOLD them that it was going to be a problem. WE TOLD 'em!!"

(CYA)^5 = Press releases from now on

Night train

-- Jes a tired ol footballer (nighttr@in.lane), December 03, 1999.

The following is an excerpt of my December Comments and Impact Scale Rating. A longer version of the comments is available at http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=001un9 The entire set of comments can currently be found at my website at http://www.gwu.edu/~y2k/keypeople/gordon. Once there, click on "Comments, Essays, & Op-Ed Pieces".

While I completed these comments before the current thread was posted, they seem to address many of the issues and questions that are raised in the thread. Rather than put together something new, I thought that I would send these excerpts along. I hope they are helpful.

I also note that today on Roleigh Martin's listserv, there is an item from Mark Frautschi that addresses some of the same issues: Item #1653 at http://www.eGroups.com/list/roleigh_for_web/?start=1653.

December Comments and Y2K Impact Scale Rating (Excerpt)

The Presidents Councils Perspective Changes on Embedded Systems:

In the latter part of October, I talked with two individuals who have been in close touch with the President, the Vice President, and/or the head of the Presidents Council. I am still trying to fully absorb what these two individuals shared with me concerning what is happening and why. What I can say is that it appears that I have given Mr. Clinton too much credit for his level of understanding of the crisis that we are in as a result of Y2K and embedded systems. While it seemed to me that he must have been at least at a personal 3-5 on the impact scale, I now think that it is quite plausible that he is no higher than a 2 or 3. I have the sense that he is, for all intents and purposes, essentially disengaged from the problem known as Y2K! He is truly leaving it in the hands of the person who head the Presidents Council on Year 2000 Conversion. The President is relying on the Chairman of the Council to let him know the status of Federal efforts and whatever else that the Chairman feels that he should know.

[It should be noted that while the President is disengaged from Y2K, he is however apparently deeply concerned about other threats to national security involving cyberterrorism and other forms of terrorism that could also be manifested in the coming few years. Indeed, I believe that it is the case that many of the actions that people are attributing to clandestine preparations by the government for Y2K are actually preparations to defend against various forms of terrorism.]

On November 9, 1999, the Presidents Council and the Office of Management and Budget convened a meeting involving a small group of embedded systems experts. The result of that meeting was reflected in part in a press release that was issued by the Secretary of the Department of Commerce. On the same date, the National Institute of Standards and Technology issued an article that focused on embedded systems issues. The Secretary of Commerce urged that efforts need to be redoubled to test for year 2000 computer problems that are hidden away in a variety of machines other than computers. See http://www.nist.gov/y2k/embeddedarticle.htm and

http://www.nist.gov/public_affairs/releases/g99-204.htm The Chairman of the Presidents Council was questioned about the November 9 meeting at the Press Briefing held on the occasion of the release of the Councils Final Assessment Report at the National Press Club on November 10. A New York Times reporter wrote the following of the exchange that he had with Mr. Koskinen after the formal Press Briefing had concluded.

"Another concern, which Koskinen said he was briefed about on Tuesday at an Office of Management and Budget meeting with computer specialists, is that some computer systems that do not appear to track the date may nonetheless have date-sensitive microchips in them. Those systems also have to be tested and plans must be made to handle breakdowns, Koskinen said." From: http://www.nytimes.com/library/tech/99/11/biztech/articles/11year.html

According to an embedded systems expert who is acquainted with Mr. Koskinens change in perspective on this issue combined with my own knowledge of what was determined at the November 9 meeting, the quote should more correctly have read:

"Another concern, which Koskinen said he was briefed about on Tuesday at an Office of Management and Budget meeting with EMBEDDED SYSTEMS specialists, is that some EMBEDDED systems that do not appear to track the date may nonetheless have date-sensitive microchips in them. Those systems also have to be tested and plans must be made to handle breakdowns, Koskinen said."

I would add these major and continuing concerns regarding embedded systems failures. The first is from my Part 2 of my White Paper:

When embedded systems fail, they can fail in a variety of unpredictable ways. Small, seemingly insignificant failures can trigger other system failures." [From Page 40 of Part 2 of my White Paper: "A Call to Action: National and Global Implications of the Year 2000 and Embedded Systems Crisis". See http://www.gwu.edu/~y2k/keypeople/gordon.]

I would also add that the timing of the triggering of other system failures cannot be readily predicted since the environment in which the failures are taking place is dynamically changing. Once the failures have occurred and have triggered other failures, the root causes of the initial failure can be hard if not impossible to determine.

Understanding embedded systems is crucial to understanding the crisis nature of the situation that we are in. The absence of understanding of embedded systems has played a major role in the governments approach to addressing Y2K. In my view, the failure of the Administration to recognize from the outset the importance of consequences of the malfunctioning of embedded systems has resulted in an extremely flawed approach to addressing the problem and a failure understand its complexities, along with a failure to recognize the crisis nature of the problem.

The Presidents Council has failed to give adequate attention to the highest risk, highest hazard systems, plants, sites, pipelines, facilities, etc. The Presidents Council has failed to take the action that it should have taken to help ensure that impacts that can be expected as a result of malfunctioning embedded systems in highest hazard, highest risk sites, plants, facilities, systems, pipelines, refineries, etc., etc. would be minimized to the extent humanly possible.

Even with the late recognition concerning the seriousness of embedded systems problems as of the November 9 meeting, no major initiatives involving embedded systems have been apparent on the part of any agencies or departments of the Federal government apart from the statement of the Secretary of the Department of Commerce. The important implications of the November 9 meeting and the subsequent press release and article at the NIST website, seem not to have been recognized or shared with the President, the Secretary of Agriculture or the Secretary of Energy, based on remarks they have made since the November 9 meeting.

(End of Excerpt)

-- Paula Gordon (pgordon@erols.com), December 03, 1999.

The goddamn corporations are responsible for their own damn embedded systems, and they don't like the gubmint telling them what to do anymore than we do.

Excuse me, but if the freaking CEO of Exxon oil is smart enough to take home $32 million in one year for his brainpower, then I think he should be smart enough to know when his business is at risk without any help from John Koskinen or Clinton.

If those greedy bastards chose to ignore computer programmers and electronic engineers because they didn't want to spend the money, then I hope they come apart at the seams. Nothing would make me happier than to see the oil industry get f**ked up beyond all recognition, clearing the path for altenative forms of energy and transportation.

-- Hawk (flyin@high.again), December 03, 1999.

Check this out

Down with Big Government, up with BOYCOTTING!

-- Hawk (flyin@high.again), December 03, 1999.

...................flying high again!!!!

-- whowinsinadepression? (karlacalif@aol.com), December 03, 1999.

Dear Hawk,

I tried to spell this out in Parts 1 and 2 of my White Paper (http://www.gwu.edu/~y2k/keypeople/gordon): Y2K and embedded systems are presenting new challenges unlike those that have ever been faced before, nationally or globally. At this point in history, it is becoming apparent that we have not kept pace with technology. It has also become apparent that people have cut corners and been shortsighted. It is obvious now that adequate care has not been taken to avoid problems and even serious failures.

In a very real sense, technology has gotten out of hand. Thoreau had pointed out that man was on a path that could lead to his becoming a tool of his own tools. Most people seem oblivious of either the threat or the emerging reality. Our tools have at least temporarily gotten the upper hand.

We are living in a time when people have become so specialized in their knowledge that those who focus on one speciality cannot begin to understand what people in other specialities are saying or doing. This is too often the case when it comes to CEO's or managers and CIO's and information systems specialists. It is no less the case when it comes to politicians or public administrators and IT specialists and embedded systems experts. Indeed, it is more often than not the case that IT specialists operate in a world apart from embedded systems specialists.

It might help to illustrate some points by using an example that is quite divorced from the Y2K and embedded systems crisis. Let's say we had two simultaneous epidemics that have spread throughout the world and that there were only a limited number of specialists in the world who knew what the remedies were for one of the diseases and that there were even fewer who had the necessary knowledge and skills to help stop the spread of the other disease. Let's also say that of those who had the knowledge and the skills concerning either disease, only a portion were inclined to devote their time and efforts to taking the needed action to quell the epidemics. The challenge would be to stop the epidemics using the limited expertise available. Let's say that there would be no way for the efforts of these individuals to effectively stop the spread of the two epidemics without some additional help from the public and/or the private sector. Additional help might include administrative expertise, help in mustering and orchestrating needed resources, help in maximizing available expertise and assistance in making expertise and resources widely available, help in identifying and applying the best and most effective remedies, and leadership that would help ensure that efforts were carried out.

The effectiveness of efforts could require Churchillian leadership, leadership that would help people understand what needed to be done and would motivate them to do what needed to be done.

Where would the resources come from? Where would the administrative and technical expertise come from? Where would the leadership come from? What if persons in key roles of responsibility in the private sector failed to understand the seriousness implications of two epidemics that had spread? What if the private sector failed to come forward with the needed resources, expertise and leadership? That would seem to leave the public sector.

What if persons in key roles of responsibility in the the public sector lacked an adequate understanding of the seriousness of the problem? What if the administrative and technical expertise was lacking? What if the political will was lacking to address the problem as it should be addressed? Whether or not the problem was adequately comprehended by the public sector, if the private sector failed to take the action needed, it would be incumbent upon the public sector to take action. Indeed, the government of this nation has an obligation to ensure that the interests of the people and the nation are protected. Surely we would not expect the government to stand idly by when it could help to orchestrate actions and bring resources to bear that could stop two epidemics that were threatening the overall health and welfare of the whole society.

With the Y2K and embedded systems crisis, it appears to be the case that some of the persons in key roles of responsibility in the Federal government have not comprehended the seriousness of the situation that the nation and the world are in. It also appears to be the case that administrative and technical expertise are lacking. Political will that is needed to address the problem as it should be addressed is also lacking. Nonetheless, it seems to me that our best hope is to call upon the government, indeed, demand that the government apply the nation's resources to helping the rest of the public and private sectors address the crisis before us. It is incumbent upon the Federal government to address this set of challenges and threats which, if left unaddressed, could have ruinous consequences nationally and globally.

I have pointed out in Part 3 of my White Paper what the Federal government should be doing. I have described in Part 5 what our options are if the Federal government fails to recognize and fulfil its obligation to the nation.

The Y2K and Embedded Systems Crisis is not a matter that has precedents. It is an extremely complicated set of problems that are converging within a given time frame. The understanding of the technical aspects of this set of problems are understood by only a small fraction of the population. Very few of the people who understand the technical aspects also understand the societal implications, and very few of those who understand both are in positions of public OR private sector responsibility. We are in a very difficult situation. To complicate matters, there are people at the helm who don't realize that they don't have an adequate understanding of the problem and they do not seem to recognize that what actions they should be taking.

There are also people in key roles of responsiblity who are not acting with the sense of responsibility that one would hope all who served in public life would have.

The bottom line is that people in key roles in the private sector have not taken the lead to date. The public sector is not doing all that it could and should be doing. We are in trouble, big trouble. The leadership is lacking and people in key roles of responsibility have not acknowledged the trouble that we are in. They have not dealt directly with the problem in the kind of comprehensive way it should have been addressed and they have not helped the public understand the seriousness of the problem and what measures the public should take to prepare for any of a range of disruptions that are likely to occur.

Trying to help educate people when they are already on the job is a big order, but it is something that can be done. Trying to help educate a public that is largely uninformed, ill-informed, or misinformed is a huge challenge, but it can be done. These efforts will not be for naught as there is an infinite amount that can be done, both now and after the rollover, to prevent and minimize our losses. Our greatest challenge is to begin from where we find ourselves at the present moment and try to do the best we can to improve our current situation. The past cannot be undone; we have to try to move on from here without wasting energy on blaming or damning those responsible for bringing us to this point. We have to keep our eyes on the prize of getting through these next months with as little loss of life as possible, with minimum impacts on public health and safety and the environment, and with the social fabric in one piece.

-- Paula Gordon (pgordon@erols.com), December 03, 1999.

Gee, do you suppose the media will print a word of this?

Amazing how the press seems to hang on Koskinen's every word, how he seems to always be there to quash the latest "leaked document", how he's gone out of his way to walk that wonderful tightrope between preparation and panic in as public a way as possible.

Now that this memo has made the internet rounds, what (if anything) will the media say about it? My guess: squat.

Still waiting for someone, anyone, to "debunk" this.

Prepped but getting nervous...

-- Steve (hartsman@ticon.net), December 03, 1999.

Paula:

I, along with countless others, truly appreciate your efforts. However, I believe the time has come to acknowledge that your efforts, and the efforts of many other Y2k activists, has effectively come to an end.

The time for leadership has passed. Lines have been irrevocably drawn in the proverbial sand. Clinton has addressed the nation for the last time on the subject until the rollover, IMO.

I understand your desire to continue the fight, but when the fighter's getting beaten to a bloody pulp, someone needs to throw in the towel.

That time has come.

Or has it??

-- Steve (hartsman@ticon.net), December 03, 1999.

Paula,

I agree that some technology has gotten out of control, but there comes a time when we should let it die. Technology has its own sort of "process of natural selection" like everything else under the stars. There is "good technology" and "bad technology." The use of fossil fuels may have served us for a while, but it is bad technology, and is destined to either fail or kill us.

I think we are seeing a sort of karma of the universe occuring here in the fact that the embedded systems in the oil industry now seem to be in jeopardy. Software systems are more easily remediated because they still serve a useful non-destructive purpose in our society. This is the way it was meant to be.

Government is really only needed for very few functions, as the laws of nature and economics will tend to maintain balance in most areas. Part of the problem with government is that they are already doing too many favors for corporations at the expense of the taxpayers. I can't see asking the taxpayers to pour more money into greedy corporations who are already getting exorbitant amounts of corporate welfare and taking very little responsibility with regard to the future health of our planet.

-- Hawk (flyin@high.again), December 03, 1999.

RC and Paula and just another engineer too,

I have never mentioned a thank you for your work, and your posts here, which has kept me informed and helped keep me motivated over this past year. Thanks.

RC, or maybe Linkmeister, (he/she is up on everything!) could bring RC's series of posts out of the archives, on a collective thread, similarly with Ms. Gordon's and just another's writings.

If a person couldn't put two and two together based on this, and the above mentioned writings, well, that is just sad.

But I do believe we have a new group of newbies here, and they deserve access to this information...it was pushed into the archives to fast by the trolls. Otherwise, newbies, search out the writings of RC and Ms. Gordon and just another engineer, you won't be sitting on that fence for long.

Best to all, keep preping, you are not done...it is not time to stop yet.

-- Lilly (homesteader145@yahoo.com), December 03, 1999.

Night Train:

I know from experience what happens when you produce a study for the government that doesn't say what the government wants -- it never gets released, and you never get hired again. I think we can be quite certain that Century "found" exactly what NIST wanted them to, just from the fact of release.

The real questions are, why did NIST want to say this, and why so late? The best guess I can come up with is that it's CYA just in case -- it was released without enough fanfare to penetrate public consciousness, but with enough so they can point to it later, if necessary, and say See, it wasn't OUR fault, we TOLD you to test better! And if problems are minor, almost nobody will even remember that study. Besides, the study didn't mention actual problems, it just said there *might* have been "hidden" problems. A win-win situation for NIST, I think, perfect for taking credit and deflecting blame.

This statement sounds like more of the same. Testing every last embedded system end to end has never been remotely feasible -- intelligent shortcuts have been an absolute necessity. The task has never been the impossible one of eliminating risk altogether, the task has been to keep risk to a minimum within the bounds of what's possible. But if minimum risks turn out to be too high, well, WE told you to do the impossible and you didn't do it, so it's YOUR fault.

I'm feeling cynical today, but I don't see any other good reason for statements like this.

-- Flint (flintc@mindspring.com), December 03, 1999.

Very important thread.

To the top.

-- cgbg jr (cgbgjr@webtv.net), December 03, 1999.

Um Shakey?

I know I've asked before, but if I bring my water, food, and ammo, is there any room left in your bunker?

Got water? food? gold? ammo? lime?

-- nothere nothere (notherethere@hotmail.com), December 03, 1999.

Flint,

There is alot of background on events or milestones leading up to the November 9, 1999 government-convened meeting on embedded systems. This background has not been reported on in any public forum that I know of. I will share a little of that background with you here.

The Chairman of the President's Council asked the National Institute of Standards and Technology (NIST) for some clarification concerning the whole subject of embedded systems sometime in April or May. I first learned of the details of this in May.

Dr. Gary Fisher has been the key point person at NIST on that effort. He began work on a draft report.

Mr. Cherry came on the scene sometime in July. Mr. Cherry contacted me and I invited him to speak at the George Washington Y2K Conference that was held the last week in July. I also suggested that he get in touch with Dr. Fisher. He got in touch with Dr. Fisher and has obviously been helpful to Dr. Fisher in putting together the paper that formed the basis of discussion at the November 9 meeting.

As I understand it, a paper was not released earlier owing to a desire on the part of NIST and the President's Council to make sure that the paper did indeed reflect a consensus of experts in the field. It is not quite so cut and dried as that, but that figured in the delay.

Since July, Dr. Fisher has co-authored work on embedded systems with Michael Cherry. It is on the NIST website.

I think if you talked to Michael Cherry of Century Corporation, you would find very quickly that he is not only not cooptable, he is extremely knowledgeable and extraordinarily effective in getting his points across. Others sharing similar attributes also took part in that November 9 meeting and also deserve similar credit. Together they have helped to give credence to the views of those who have first hand experience working in the field.

Those interested in learning of the perspective that Dr. Fisher had on the subject in July can view a video of a panel he participated in at that time. You will be able to judge for yourself how much or how little his perspective has changed since his colloborative efforts with Mr. Cherry began. Dr. Fisher participated in a panel on embedded systems at the July Y2K Conference at George Washington University. Dr. Mark Frautschi was also a participant in that panel. For information on how to obtain a copy of that video, see my website at http://www.gwu.edu/~y2k/keypeople/gordon. Click on Announcements. If you have any trouble finding the information, please feel free to e-mail me.

I think that you will see that Dr. Fisher exhibited considerable understanding of this complicated subject matter.

Michael Cherry had been invited to take part in that panel. At the last minute, neither he nor a colleague of his from Century Corporation were able to take part. (Both, however, were at the November 9 meeting.) Mr. Cherry got in touch with Dr. Fisher sometime in July. Their colloborative efforts began after that.

I hope this clarifies some of the background here.

-- Paula Gordon (pgordon@erols.com), December 03, 1999.

Ouch. Still, it's worth pointing out that all of the basic points made here were made at least 18 months ago (to my knowledge) in Dr. Frautschi's technical, online paper on embedded systems. In other words, while this may be news to Koskinen and friends, it's not news to folks who have been following this particular issue.

It's also worth remembering that embedded systems that don't have an evident date-keeping function but that have a hidden date-keeping function are considered to be fairly rare. (Peter de Jager earlier this year also had an excellent essay on this general subject, entitled "A Crock of Clocks.") That doesn't mean there might not be a few of those systems out there waiting to cause some significant trouble--it's the one you don't see that gets you!--but it's not exactly a widespread, endemic problem.

I'm more concerned about the general lack of comprehensive, end-to-end system testing; that might spell trouble with some of the large-scale embedded systems. Since none of this is my field, I've just tried to educate myself as a layman on the issue; unfortunately, I haven't seen much detailed, technical analysis and discussion of such systems and possible Y2K threats to them. (I'm thinking here about Distributed Control Systems and Supervisory Control and Data Acquisition systems in particular.) One wonders how well the interfaces in and among such systems are being checked out. It's my understanding that, at least in the power industry, it's often possible to work around or without SCADA; that may not be the case with DCS in generating plants.

It's good to see Koskinen acknowledge openly that adopting an FOF policy with regard to embedded systems could be a recipe for serious trouble. Unfortunately, my understanding is that quite a few companies (especially the small and medium-sized ones) in the U.S., and a significantly larger percentage of companies abroad (maybe most of them), are adopting an FOF policy. I'm talking here about companies in general, not the power industry per se, please note. I doubt that very many U.S. power companies have adopted an FOF policy, at least to any significant extent. Power company engineers are dedicated people with families to think about, too, you know.

The late Harlan Smith (one of the mentors for Dr. Frautschi's paper, I believe) once remarked that the embedded systems problem isn't as large as originally feared, but it's still larger than we would like. That seems true. And unsettling.

-- Don Florence (dflorence@zianet.com), December 03, 1999.

forwarded: Recent Discussion of Embeddeds by an Expert

-- Ashton & Leska in Cascadia (allaha@earthlink.net), December 03, 1999.

Absolutely true, Don. Frautshi, Beach, Paula Gordon and others have brought this up repeatedly over the past year at least. But now, it's official, so it's OK to believe it.

Up yer ante, Kosky, you pompous ass.

Goodnite, Irene.

Godspeed,

-- Pinkrock (aphotonboy@aol.com), December 03, 1999.

The bozos at my local Y2k forum seem to be convinced that this memo is a hoax. Can we silence them by proving its veracity?

-- Steve (hartsman@ticon.net), December 03, 1999.

Anybody have Koskinen's e-mail address? Let's just ask him. The memo says feel free to disseminate. I think it's real.

-- Dog Gone (layinglow@rollover.now), December 03, 1999.

We have Koffinsky's addy but would we use it? NO! BBWWAAHAHHAHAHAHAHAHAHAHAHAHAHA wouldn't trust ourselves that far, we'd get arrested! Heeheeheeheeheeheeheheheheheheheh. Restraint, restraint, gotta run now.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), December 03, 1999.

Perhaps Snyder, Cowles, or Factfinder can give us some verification as to the authenticity of this memo.

-- Paul L. Hepperla (paulhep@terracom.net), December 03, 1999.

This one needs to stay at the top, friends.

-- Spindoc' (Spindoc_99_2000@HeadDown.now), December 04, 1999.

None other than cpr verifies the memo's authenticity here.

-- Steve (hartsman@ticon.net), December 04, 1999.

Close tag. Link works. cpr goes by "buytexas" on my local Y2k forum. He's been roused from his cave...

-- Steve (hartsman@ticon.net), December 04, 1999.

Steve,

I'd try searching info.gov, for confirmation, since it would most likely show up there, but... of course... the search engine is down at the moment.

Aaaaaaaarrrrrrggggghhhh!

*Sigh*

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), December 04, 1999.

More than one way...

;-D

Subject: [civicprep] FW: Position change relative to embedded systems....
Date: Thu, 2 Dec 1999 20:39:52 -0500
From: "Steve Davis" < steve@davislogic.com >
To: "civicprep" < civicprep@4hlists.org >

Here is information from John Koskinen on the embedded chips issue. I think it relates directly to the "Mr. CEO" issue and chips clock issues.

Steve Davis
http://www.DavisLogic.com/
http://www.Coalition2000.org/

-----Original Message-----
From: John_A._Koskinen
Sent: 02 December, 1999 12:31 PM
To: steve[snip]
Subject: Re: Position change relative to embedded systems....

For your information, here's a cover memo and summary document on the chips meeting that we've sent to the President's Council, the Federal CIOs and our private sector working groups. As you'll see, at the meeting the experts from electric power, telecommunications and oil and gas all agreed with the statements in a context in which they were comfortable that they had taken these steps.

All the best.

(See attached file: chips meeting final no red.doc)(See attached file: chips meeting cover memo.doc)

###

Steve,

Im just gonna forward the attached files to you.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), December 04, 1999.

Thanks, Diane. My question from earlier remains: What (if any) media attention did this receive? Why the collective silence on such a vitally important piece of information?

Has anyone seen any media outlet pick up this "non-story"? Methinks Koskinen, while apparently encouraging the dissemination of the memo, has simultaneously squelched media coverage.

Once again, managing to walk that tightrope between preparation and panic, information and ignorance...

-- Steve (hartsman@ticon.net), December 04, 1999.

To Don Florence,

Your comment about there being so few embeddeds involving dates as to be nothing to worry about... The Kosky point does not do that. Further more the NIST/Century Report states that it is 'far worse' than ever previously thought. So too for Jim Lord's "Mr. CEO"... My multiples of sources handling the embeddeds situation in the oil industry tell me that the embeddeds problem is a literal nightmare and that there are a lot of embeddeds out there without obvious date factors that are indeed still linked into date processing. Worse yet, many of these systems (esp. older ones) have lost all design documentation. No one knows what the hell is going on with a lot of their oilwell, pipeline and refinery systems. So these reports are indeed important and confirm what my sources in the field have been telling me all along.

-- R.C. (racambab@mailcity.com), December 04, 1999.

Steve

The DOC issued a press release that made it on the forum. The press release is based on the NIST paper which is also on the forum.

Now how many folks even on this forum understand the implications of this? Then figure the press would understand?

And the public??? Ha not a chance, they still think we are going into the new century next year. Something so simple can get fucked around as the "date" of next year, how the hell are they supposed to understand embedded systems.

I have been reading stuff for the last two years from "engineers" or what ever disputing what Koskie has just confirmed. And they are supposed to be educated?

Even Flint should be able to see the writing on the wall now.

Fence sitting doesn't get you anywhere. Didn't this time. In this case ignorance isn't bliss. It is ignorance plain and simple.

-- Brian (imager@home.com), December 04, 1999.

Brian:

Koskinen has been a laughingstock of this forum since its inception, for being ignorant and evil. But suddenly he notices that embedded systems might have bugs, and the writing on the wall suddenly changes completely -- Koskinen is now the fountain of truth, and all those so- called "engineers" were lying! We KNOW this, because we're now armed with a report from a company that tests for and fixes embedded bugs for a living, saying embedded systems have bugs and need testing. Who could be more objective than that?

Can you see how silly you look?

-- Flint (flintc@mindspring.com), December 04, 1999.

The NIST report is terrifying, but it is a fact that, on December 4, 1999, EVERYONE is blowing smoke out their rear ends.

It is amply alarming that 27 days before Happy New Year, we still don't know credibly what will happen with embeddeds. Excuse me?

"RC", "Gecko" and some others here, at least in oil, have long since passed my "smell test". But why should I be relying on "smell" at this point?

Koskinen is an idiot. So is Commerce. Flint is right about that one. They are no more reliable on this than they've been with their other pronouncements.

That hardly is reassuring, however ..... my entire reason for "pessimism" all along has been the mix of incompetence, PR/spin and sheer "deliberate misleading" that has characterized Y2K from the beginning (a beginning which initially featured the "this is a hoax" line as the first "mainstream spin", if you're old enough to remember).

Got Murphy's oil?

-- BigDog (BigDog@duffer.com), December 04, 1999.

I dont wish to sound 'dumb' but I have read this thread, and flint, I for one, dgi...I see that many here are upset, but can anyone tell me what is between the lines so to speak? Does this mean that there will be explosions because of pipelines as someone here said? I'm sorry I dont understand, but as for computers, I only know how to use them. Sign me.....soooo confused and sorry.

-- consumer (shh@aol.com), December 04, 1999.

The slight delta shift here in Koskinen, from happy face to "guarded concern," is what's important to note, Flint.

GI?

Or not.

Wouldn't trust him with a ten-foot pole. But when a pattern shifts... well... pay attention to the subtle stuff.

Above all... completing preps is a good idea.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), December 04, 1999.

My first thoughts on this were--to paraphrase the favorite saying of Jr. High School students--"like, DUH!"

Nodes in a system are interconnected. Just because a node doesn't have an overt date stamped on it DOES NOT mean that it never processes a date, or never processes a date INCORRECTLY. And I am not even good at computers. Why didn't people catch this? Are they too drunk or stoned to notice?

I am not very hopeful if Mr. Sunshine, himself, Koskinen, says something like this at the last minute. For all I know we could end up as Milnetoast, now. I would like to personally thank you for alerting us of this news and casting a gloomy shadow over my day. ;)

-- coprolith (coprolith@fakemail.com), December 04, 1999.

Diane:

Yes, some embedded systems have date bugs. Some few of those date bugs are surely serious. The first weekend in January should be surreal. And the collateral damage caused by some of those bugs will take months to clean up, IMO.

But neither the incidence nor the impact of those bugs changes whatsoever, regardless of any "shift" in Koskinen's PR. And no report detailing the mechanism or root causes of the bugs increases their number either, any more than a detailed explanation of what *might* happen if you roll snake eyes increases the probability of doing so.

It's good to see official pronoucements move closer to recognizing the reality. But they don't change the reality itself by doing so. They only stand to increase the number of people who can see the prudence of taking precautions. This is a Good Thing.

-- Flint (flintc@mindspring.com), December 04, 1999.

A Comment from Flint

Can you see how silly you look?

-- Flint (flintc@mindspring.com), December 04, 1999.

Well what ever happens to Koskie doesn't consern me. There are mantras by folks that say they know about embedded systems that seem to be wrong. This goes back to when I was lurking on CSY2K from almost 2 years ago that have been disputing the problems with embedded systems. The document that Koskie has put his name to has clearly sided with many of the pessimist embedded systems commentators such as Tim May.

It is true that I have painted the engineers with a broad brush and I feel a bit silly about that but if the problem was clearly understood by all parties as of last year then there wouldn't be the indecision at this late date about the status of remediation. Clearly there still is this indecision and it blows my mind that it still exists.

Of course if society continues to operate in an exceptable manner then yes I am feeling pretty stupid. But we should find out during the first week of January the extent of the problem and if Koskie is in CYA mode then there should be alot of other silly looking folks out there.

Flint you know damm well that this is the clearest warning issued in North America on the topic. And we are reading and writing about this topic in Dec. 99. You know this should have happenned last year. If you could explain why it didn't then I am all ears.

For example Dale Way of the IEEE saw little consern about embedded systems, now he maybe right, and I should keep my mouth shut. But then that would make this statement useless. If you can explain why the NIST would raise conserns that the IEEE Y2K chairperson disregards, I would appreciate it Flint.

There is something here that doesn't add up.

-- Brian (imager@home.com), December 04, 1999.

Brian:

What I think doesn't add up for you, is that the number of different *ways* an embedded system can screw up, is entirely distinct from the number of different *systems* that will screw up. You appear (at least to me) to fall into the error of believing that the longer the list of potential failure modes we can compile, the greater the danger. As I keep saying, doubling the size of the list of things you can run into if you fall asleep at the wheel doesn't change the *likelihood* of falling asleep at the wheel whatsoever. You seem to have fallen into the error of thinking that if the list is twice is long, so is the probability. T'aint so.

One major conceptual problem we have with embedded systems is that they come in so many flavors. When you build a normal computer, you are quite restricted in your design, since you want that computer to run popular operating systems, accommodate popular peripherals, and use existing software. But when you design an embedded system, quite often you have the freedom to start entirely from scratch -- your imagination need not be fettered by standards, existing protocols or installed bases. You just invent these as you go along.

The Century/NIST report simply adds to the list of possible ways things might fail. And I agree they have identified one such way ("hidden" date functionality despite no obvious use for the date) which CAN be done, so it surely HAS been done. But our exposure is no greater AFTER you learn of this than it was before. The danger hasn't been increased, only your *perception* of danger. The fact of release of the Century/NIST report is partly PR and partly advertising.

-- Flint (flintc@mindspring.com), December 04, 1999.

Actually Flint

If you could be so kind, please point me to a paper from a US site that discusses the above issues (from the NIST not Koskie) and is refferanced by members of the engineering "guild" and I would be glad to feel very stupid. (Like the GM remediation guidelines)

And the IEE site is in England so that doesn't count. And the site should have been up from last year. You must know of one or two eh?

Help me out here Flint, all of these problems should be clearly understood and well documented somewhere on the net by a US source. If there isn't one then why is that?

(Mind you there is the Mitre PDF file that is interesting from late spring of this year.)

-- Brian (imager@home.com), December 04, 1999.

R.C.

Actually I was saying that embedded systems with truly hidden date functions are supposedly relatively rare, based on reports I've read in the past. I wasn't commenting about Y2K in embedded systems generally (except in my closing reference to Harlan Smith). But you're right--the original (Nov. 22nd) NIST/Century Corp. report does suggest that there may be a lot more of these hidden problems than previously believed. I went back and checked. I have no idea who is right here; as with so much involving Y2K, it seems that you pick your "experts" and you take your chances. That's why it all seems so unsettling.

I was pointing out that the technical basis for Koskinen's memo (however much folks may argue about the frequency of hidden date problems, RTCs, etc.) was laid at least 18 months ago in Dr. Frautschi's paper.

Finally, as noted before, if there are serious troubles, I suspect they will arise in the large-scale embedded systems, in the ways that date fields/formats are processed and passed through those systems. Many of these systems, like DCS and SCADA, are also usually PC-based (i.e., are not just dedicated, firmware systems), typically of the x86 variety, which increases the potential for problems. And, as you know, there are a lot of such systems used in the oil & natural gas industries.

Bottom line: yes, I understand your concern.

-- Don Florence (dflorence@zianet.com), December 04, 1999.

"Can you see how silly you look?" -- Flint

AHA! Looks like it was Flint trolling as (do you see how stupid/silly you look).

-- whaddya know (everybody@has.a dark side), December 04, 1999.

Flint mentioned

Brian:

""What I think doesn't add up for you, is that the number of different *ways* an embedded system can screw up, is entirely distinct from the number of different *systems* that will screw up. You appear (at least to me) to fall into the error of believing that the longer the list of potential failure modes we can compile, the greater the danger.""

NO that is not what I am talking about. The issue I am raising is awareness of the risks to embedded systems not the risks themselves. Alot of the NIST paper has been mentioned before on alot of different sites but no American site seems to have had a definitive assessment of the risks that can be refferanced by anyone that needs the info. Such as the IEE site.

It is not the increased risk of failure, nothing is going to change that. All I would like to see is a site in North America that details the issues raised in the NIST document.

For example a local food store chain had hand held scanners that had no date input and no reason to know the date. It was tested and failed the rollover. How many companies are going to be as thorough??

They said that is would have impared the way the business functions. Now how are folks such as the food store going to have known that this scanner is a risk by the general info on the net about Embedded Systems?

Flint we are on the wrong wavelength on this topic, for me it is a matter of general awareness not another sign that the world is ending. I don't think the world is going to end. But I do think that the general awareness of the risks has been dumbed down so that most would not understand them as they really are.

At least that is how I see it. How are average businesses going to know that there is a potential problem if they don't have a clear understanding of the risks?

-- Brian (imager@home.com), December 04, 1999.

Brian:

I wish I could help you out more than I can. I know that more Electronic Engineers work in embedded systems than in every other area combined. There is no database or clearinghouse or other central location where such systems are archived, except as we build such things piece by piece. There is no protocol in place for any kind of central registry or documentation of such systems. We design them, we build them, we sell them. In mind-boggling variety and number.

I'm curious when you write:

[For example a local food store chain had hand held scanners that had no date input and no reason to know the date. It was tested and failed the rollover.]

But how was rollover tested, other than by setting some clock ahead? In which case it was not "hidden" in any way. I always wonder about tales like these. I don't doubt there was such a failure, but the "dressing" confuses me. Why was it tested if nobody saw any reason to do so? In what way did it fail?

To me, these questions are important because of the wide variety of systems we use. You simply can't generalize usefully. Every last damn system must be considered, case by case, detail by detail.

I'll agree that we can improve the accuracy of our tests with more knowledge of the details of the system being tested, and we can improve the functional coverage of those tests. But I think this is well down the priority list. At the top, we need to examine the most critical systems as carefully as possible. And we need to examine the actual failure mode, to see if remediation is cost effective. The above story said this:

"Estimates ranged from a 1 - 2% potential failure rate of processes containing embedded systems in some sectors to 4 - 6% in others. An important distinction was made between failure of an embedded system, which may not cause a process or device to fail in operation, and failure of a process or device due to an embedded system. ...the latter is much less prevalent."

OK, we have embedded systems failing. "Much less" often, this causes a failure in the device or process. Much less often than that, the device or process failure has an impact on the business. And much less often than THAT, the business impact is large enough to become visible to the public. It's a rare failure indeed to seriously injure a business. We know which processes, if they break, can do this. We examine those processes as well as we can. Yes, we'd like to be armed with as many different things to look for as we can find. But knowing *what* to look for in general is very different from knowing *how* to test this particular device. Everything's in the details.

-- Flint (flintc@mindspring.com), December 04, 1999.

all right Flint we are on a roll here. Remembering that I am a non Tech kind of person that respects the business below and their efforts to make sure I can still get food from their store :o) (which I do).

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

 Thrifty Foods Tackles Monumental Y2K Project

"Every piece of hardware, software, or embedded system," Thompson
explains, "either underwent a test script or has a vendor compliance
statement." He's quick to warn, however, against accepting vendor
compliance statements at face value, citing the stores' 60 hand scanners
as an example. These handheld devices are used for ordering products
and maintaining inventory. As such, they're an absolutely essential part
of Thrifty's operation.

These devices appeared to have no date functionality at all and Thompson
had a vendor compliance statement on file. As an added precaution, he
decided to test one of them. Much to his surprise, the BIOS failed.
No-one realized that these devices had internal clocks. Had they missed
this, these innocuous little appliances could have seriously impaired
Thrifty's ability to function smoothly on January 1, 2000.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Now this appears to me to be important to the function of the business. How are they going to know or trust anyone if you can't believe the vendor or the company that tests the scanners? That is what seems to be the case. If the scanners failed in a benign way that didn't impair the needed function (gathering info on inventory)  then why would they have to be replaced? I am sure that Thrifty Foods has got better things to spend their money on than 60 new scanners?

If they did fail and could not gather the information the store needed then how could this have happened? How could the date function impede the other functions that controlled the inventory?

How do you know that the tester isn't ripping them off or is wrong? Get a second opinion? Who is right and who is wrong? How is a non tech person to make a choice?

Thankfully in my area Thrifty's has been very proactive regarding Y2K with the provincial government. They GET IT. And they communicate the problem. This seems to be rare in the world of Y2K.

The part that really bugs me though is that some folks would accuse them of profiting from the sale of food to those that are worried. How can the "optimists" reach this verdict?

Flint if I had a nickel for all the times I have read on the net that if a system has no date functionality at all then there is nothing to worry about, I would be rich. You know that.

They are obviously wrong or Thrifty's got bad advice. Somewhere along the line this is really screwed up.

By the way the article was from the end of March 99. I have know about this since then. If I wanted to be a Doomer I would have brought up this topic sooner but there has been little information to confirm that this is a problem. And I like confirmation.

Waiting for 8 months for some document to confirm this???

Don't think I am jumping on a bandwagon here eh?

-- Brian (imager@home.com), December 04, 1999.

TOUCHE Brian. Oh yeah we're in deep kimcheeeee for sure. 26 days.

-- thrifty (here@too.glad), December 04, 1999.

Brian:

Forgive my obsession with details, since these are what matters in such systems. Now, read this statement:

"These devices appeared to have no date functionality at all and Thompson had a vendor compliance statement on file. As an added precaution, he decided to test one of them. Much to his surprise, the BIOS failed."

The hand scanner has a BIOS? Uh, no, a PC has a BIOS. Apparently the hand scanners were plugged into a PC (it may have been a PC adapted to accomodate a lot of scanners, but it's still a PC). It was likely running DOS, and DOS was running a scanner application. The scanner application interfaced to the scanner ports via a device driver. Do I *know* this is how it worked? Of course not. This is how the hand scanner system *I* developed worked. But it sounds very similar.

So consider an exactly similar situation. Your PC has a hard drive. The hard drive is certified by the manufacturer to be compliant and use no dates. You plugged this hard drive into your PC. Your PC has a BIOS issue (not uncommon, but rarely serious). Does this mean the hard drive manufacturer's statement was in error? Nope. Does this mean you must replace your hard drive? Nope. Does it mean your hard drive made some kind of "hidden" use of the date? Nope. There is NOTHING WRONG with your hard drive. I'm willing to bet the same was true of these hand held scanners as well.

However, the system itself failed, in that the BIOS failed some rollover or 2000 date test. Yes, that's fairly common. You will notice that has nothing to do with either the physical scanners or any of the scanner software running on the PC. It's highly unlikely that the scanner manufacturer designed the PC, and there's essentially ZERO chance they wrote the BIOS. Still, Thrifty could have had a problem until they found someone knowledgeable enough to reboot the system, enter setup, set the date correctly, and forget about it. But by now, it's not hard to find people who know how to boot a PC -- Windows gives us LOTS of practice [grin].

So the issues here are (1) Where was the locus of the problem (it wasn't with the scanners themselves); (2) What exactly did the failure cause (what you quoted doesn't say); (3) How difficult was the fix (trivial); and (4) How long would Thrifty have been in trouble (up to half an hour).

Well, case by case. There's an easy one. Next. (And that's how engineers think and work. Go down the list one by one, find the problem, solve the problem, and keep going. It's the only way).

-- Flint (flintc@mindspring.com), December 04, 1999.

Message to Brian,

You asked the following of Flint: "If you can explain why the NIST would raise concerns that the IEEE Y2K chairperson disregards, I would appreciate it...."

I have a perspective on that that I thought might be helpful to share. I assume that you are talking about Mr. Way's views (the chairman of the IEEE task group who wrote the June 9, 1999 Open Letter to Congress).

In my White Paper at http://www.gwu.edu/~y2k/keypeople/gordon (click on Part 2), I have tried to clarify why some people have such a hard time grasping the significance of date-sensitive embedded systems. Believe it or not, even trained electrical engineers can have gaps in knowledge concerning date-sensitive embedded systems. In fact, a French electrical engineer that I met in May, shared with me his view that electrical engineers whose academic training was from an American University can be particularly susceptible to such gaps in knowledge. In his view the academic training of electrical engineers in the UK or France tended to be much broader.

I have communicated with electrical engineers in the United States who have little or no first hand familiarity with date sensitive embedded systems. In a way, this should not be surprising. Similar gaps in knowledge can be found in all areas of professional specialization. What you have to look for if you are searching for individuals who can speak with authority is those persons working on the frontlines who have first hand knowledge of the problems that exist and who also are conversant with the "safety critical" systems of high hazard, high risk sectors. It is not enough to rely on the knowledge and first hand experience of individuals whose experience is limited to other sectors. At least three of the non-government people that I know of who were involved in the November 9 meeting had the broader kind of background with high hazard sectors.) (That government convened meeting, as you know, also included OMB, the President's Council, and NIST.)

An aside: As far as Mr. Way's overall assessment of the impacts of Y2K are concerned, embedded systems issues seem to be far less significant to him than IT concerns. Even so, he seems to come out very high on the impact scale without a more of a concern for embedded problems.

It seems to me that Mr. Way's apparent lack of equal (if not greater) concern regarding embedded issues has to do with his apparent position regarding the stakes vs probabilities argument. He seems to be an adherent of the probabilities side of the argument. Unless a person who adheres to the "probabilities" side of the argument is aware of the higher probability of failures in some of the highest risk, highest hazard sectors, that person may not recognize that even from a "probabilities" perspective, the risks are substantial.

I am going to try to write a kind of an update on embedded issues as Part 7 to my White Paper. In the meantime I have written additional comments on embedded issues in my December Comments and Impact Rating at the same URL as above, then click on "Comments, Essays, & Op-ed Pieces." You may find these germane.

Message to Flint:

Sometimes the normal course of events in a government agency (when it comes to releasing a potentially controversial paper) takes unexpected and even unprecedented twists and turns. From what I have been able to glean about the NIST paper, this has certainly been the case. It has not taken anything like a typical path. I have tried to indicate some highlights of that process in comments I made above in this thread. I think that this is a instance when you cannot make normal assumptions concerning what might have gone on leading up to the convening of the November 9 meeting and the November 22 release of a potentially controversial position paper.

-- Paula Gordon (pgordon@erols.com), December 04, 1999.

Thanks for your time Flint

"But by now, it's not hard to find people who know how to boot a PC -- Windows gives us LOTS of practice [grin]."

Well I guess that is why I own a mac :o)

I understand what you are saying here, and it seems if you are right that the Thrifties Rep may have worded it wrong (it appears to me). It just seems so odd that they would have made such an error. Even I could reboot a windoze machine and set the date. So why would the rep. say that the scanners are at risk when they may not be? Where did he get the advice? How could the tester make the situation sound more serious that just a simple reboot? I know you don't know the answers and it is to late to worry about the issue at this point in time but it seems to raise some interesting questions.

More important to me is that the Thrifty's Rep and our .gov Y2K team are pretty tight. One of the .gov leaders is 30 IT veteran. How could he have not corrected the Thrifty's rep on such a simple plan as rebooting and setting the correct time on a PC?

I don't know but maybe I am making myself look stupid. But it does underscore my point about needing a comprehencive site dealing with this issue.

-- Brian (imager@home.com), December 04, 1999.

Paula:

I have no problems with the contents of the NIST paper. I feel they have identified a class of problems which is real, though rare, and likely to be missed without appropriate and possibly nonstandard testing. I still don't feel government agencies release studies for no purpose at all, though the process may have been complex and agreement not unanimous.

The point about American engineers being too narrow, I'm not quite following. Are you implying that these narrow engineers put those sensitivities into those systems without even realizing it? Any programmer can tell you that times and dates are nasty things to program. They don't follow any particular system. Even a task as simple as "add six months to current date" is full of exceptions. As a result, adding date functionality to code is a tedious chore, and doesn't happen by accident. And you don't forget about it either. So certainly the awareness is there among those who put the sensitivities in there. Those people are unlikely to forget about it when their task is specifically to test for date sensitivities. You'd expect the nature of that task to be a reminder all by itself.

So are you saying that people presenting overviews (like Mr. Way) have a blind spot which leads them to overlook such sensitivities? That perhaps this inexplicable blind spot among testers ("Gee, what was I looking for again? I keep forgetting") has led to an anomalously small number of such sensitivities being found during actual testing? I'm sure some have slipped past us. I hope none have slipped past the French engineers.

-- Flint (flintc@mindspring.com), December 04, 1999.