Interesting post by Just Another Engineer in a previous thread.

I work with embedded systems for real-time operating systems such as VxWorks/RTEMS/etc and networking devices for ethernet/frame-relay/etc.

Bottom line for us is - no Y2K problem. The technology moves so fast that nobody can afford to be left with redundant out-of-date techniques. One of my previous companies started off selling networking products with PROMs in the 1980s which could only be updated by physically removing the chip and putting in the one we sent them.

Now since the early 90s it is all flash-RAM upgradable - you just download the fixed firmware from the Internet, tell the unit configuration program where the new file is in on the local network and it downloads it and overrides whatever is in the resident EPROM - its a cinch.

I worked right at the hardware device driver level for two networking companies and there was absolutely no Y2K problem - there never was one to begin with - they count the time as ticks from the time of power up or in the case of UNIX-like systems such as VxWorks from JAN 1 1970.

The problem is overrated. If a chip needs to be remediated there are several routes:

1. Download the Y2K-fixed image from the provider's web site.

2. Receive the new chip by post from the provider.

3. What if the provider goes bust? How naive! The free market abhors a profit vacuum, the software is sold to a new company who negotiate a support contract with the buyers to maintain the product, they recruit one or more of the original engineers until the transfer of knowledge is completed. I should know - one of my companies did exactly this for an important customer of a defunct parent company.

4. If this fails, dump the product and buy a new Y2K compliant one.

What if the chip is totally inaccessible? Two possibilities:

a. The date change function is not critical - just disable it remotely or ignore and fix on niggly problems occuring (who cares if the logging function sends screwy dates - tell the PC or mainframe to ignore it if it doesn't already).

b. Catastrophic failure. Having eliminated virtually all chips in the above discussion just how many of these mythical mission-critical chips that no one can get to are out there? Can anyone tell me because I don't know and why put mission-critical chips in such an inaccessible area without some contingency such as a remote command system to access them? That sounds just plain stupid.

So please tell me how many of these life-or-death chips in stupidly inaccesible areas there are. If no one knows then it is all just specious speculation.

The bottom line is - there will be some problems - but not the end of the world as we know it.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999

Answers

Shuggy..........Thanks for the Post. If you don't mind me asking~ What is the expected life for a PROM type application. I realize this is rather vague because of the multitudes of uses. The reason i ask is intent may not have been mission critical intially. Another point is due to any loooooong lifespan,for what reason would someone upgrade to the new Flash-Ram that you speak of unless premature failure requiring replacement or a new application

-- kevin (innxxs@yahoo.com), October 28, 1999.

Hello Kevin,

Theoretically, a customer can stick with a PROM for virtually their working life but the demands of increasing technology, bandwidth, faster processors and new protocol features puts pressure on them to "conform" especially when the OEM limits the support lifetime of old products to 5-10 years.

In most cases, when they hit a bug or are looking to use a new feature in the technology landscape then they are virtually forced to upgrade to new modern products without old PROM technology because the new software was written for the new product range or the limited RAM/CPU power of the old unit would make it run like a dog with the new features.

It is in the interest of an OEM to encourage customers to upgrade to the newest product purely to reduce the support strain on an ever-increasing range from day on eof the companies inception.

For more proprietary systems such as public sector, the free market is less influential but in these days of big government they are forced to upgrade their systems as public sector demands increase and the need to store and retrieve bureaucratic information increases.

Also, governments tend to stick with big companies (IBM, Siemens) for these jobs because they know the support will be more stable than with a small company more likely to go bust in 5-10 years.

If they refuse to move to the Y2K-compliant chips that is their problem and not an embedded systems issue per se.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Shuggy,

Let me do some 'guessing'. You are in the late twenties to mid- thirties in age (probably mid-thirties). To work at the Device Driver level, your career has been fairly narrowly focused and very technical. You obviously work for a reasonable sized outfit, due to the fact that the product line is enhanced over time.

There's a lot more to the world of embedded processing, lot's of 'little' guys (I worked for one in about '95 that used early 80's vintage chips (at that time, they were paying about a nickel apiece for them). I know for a fact their product has not been touched in over 12 years (they weigh things, fruit, vegetables, people, trucks, rocks, cake mix recipes, ingredients for chemical processing facilities (oops).

Ticks are ticks, but do ticks drive the time or does time drive the ticks ????? Only time will tell (how ironic !!!)

-- BH (silentvoice@pobox.com), October 28, 1999.

Shuggy--

Whew! Thanks for some informed non-catastrophic thoughts. The embedded issue has been around a long time and I thought it had been rationally discounted until "just-an-engineer's" post. (which appropriately started a huge thread). Let's face it, this is THE issue. Legacy software is no-sweat by comparison. What has always seemed odd to me is that there is relatively little alarm (of which I am aware) about the embeddeds in the engineering world. If we were so hugely vulnerable here then no combination of lawyers and marketers could cover this up. (and why would they want to?).

So back to sleep I go? Not really, but at least I can go back to a semblance of quasi-optimism.

-- Lars (lars@indy.net), October 28, 1999.

~~~(It is in the interest of an OEM to encourage customers to upgrade to the newest product purely to reduce the support strain on an ever-increasing range from day on eof the companies inception.)~~~ Technology is a wonderful thing, but regardless of what the OEM thinks. The OEM rarely if ever " gives anything away". I am sure your aware of the the statement "If it ain't broke,don't fix it." With this in mind i tend to think there are many out there just ticking away. I know the company i work for doesn't spend alot of time just looking for aplications to change........ Thanks to the"Bean Counters" mainly. The information you have posted is very encouraging. I hope you more right than wrong... Kind regards

-- kevin (innxxs@yahoo.com), October 28, 1999.

~~~(It is in the interest of an OEM to encourage customers to upgrade to the newest product purely to reduce the support strain on an ever-increasing range from day on eof the companies inception.)~~~ Technology is a wonderful thing, but regardless of what the OEM thinks. The OEM rarely if ever " gives anything away". I am sure you're aware of the the statement "If it ain't broke,don't fix it." With this in mind i tend to think there are many out there just ticking away. I know the company i work for doesn't spend alot of time just looking for aplications to change........ Thanks to the"Bean Counters" mainly. The information you have posted is very encouraging. I hope you're more right than wrong... Kind regards

-- kevin (innxxs@yahoo.com), October 28, 1999.

Whoops........sorry for the duplication

-- kevin (innxxs@yahoo.com), October 28, 1999.

IEE Embedded Systems Fault Casebook

Click through the examples. Note the continual use of words like "plant shutdown", "loss of production", "catastrophic".

-- John Ainsworth (ainsje00@wfu.edu), October 28, 1999.

"3. What if the provider goes bust? How naive! The free market abhors a profit vacuum, the software is sold to a new company who negotiate a support contract with the buyers to maintain the product, they recruit one or more of the original engineers until the transfer of knowledge is completed."

I have worked on many of this high-tech acquisitions. The time frame tends to be measured in MONTHS. These are the types of arguments that bolster the potential for the economy to start turning around again later in 2000, but in no way contribute to the viability of a 3-day FOF scenario TPTB would have us swallow. You're just supporting Just Another Engineer's statements by saying these things, Shuggy.

-- Brooks (brooksbie@hotmail.com), October 28, 1999.

BH,

Not far off, I am 36 years old.

Embedded systems are a large world but we should only focus on a subset - those that need to know what year it is.

I cannot imagine why the chips you mentioned used in weighing people, rocks, trucks, etc, need to know what year it is! And remember that older chips have less RAM and CPU than modern counterparts - so the less unused code baggage the better - including useless date logic.

As for CPU ticks - I can assure you that they do not care either what year it is.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

I work with farmers and ranchers. Although mine are generally on the low tech side, there are bigger operators with fancy cold storage; climate control; harvesters; pivot irrigation systems; scales; etc that have embedded systems. Most fix mechanical breakage themselves, or with the local electrician. The machinery will probably be goosed and tweaked to last a generation. They are not going to "upgrade" on a regular basis. Some of this stuff is extremely expensive for a small operator. I know for a fact that the embedded systems issue has not been brought to the attention of many farmers and ranchers. Fortunately, many won't be farming till spring. This will help some. (Hoping climate controlled storage of last year's crops will hold and that some discover they have a problem BEFORE the time they need the equipment.)

I am sure that there are other applications like this. Not easy to just download a new program to update a chip when you have no inventory of where they are.

-- marsh (armstrng@sisqtel.net), October 28, 1999.

Kevin,

Regarding "If it ain't broke, don't fix it!" has a degree of truth, but if everyone followed that logic I would never service my car until it broke down on the highway!

IT managers know there is looming issue with Y2K but that is independent of embedded systems, that is a human psychology problem.

Those that use archaic products normally find themselves slipping down the competitive ladder to oblivion.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

John,

Regarding IEE link.

You also click and note the repeated occurences of the words "fixed", "upgraded" and "replaced" in the "What was the solution" field.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Brooks,

Regarding hi-tech acquisitions. Sure, it would take months to complete such a deal but why should that suspend support and maintenance contracts? Don't the acquirers want to keep these customers!?

And what percentage of all hi-tech firms are in a state of takeover at any given time - a minority.

Alos, I do not advocate fix on failure. My reason for posting was to point out that embedded systems are not some dragon that is too big to slay but that doesn't mean managers can rest easy and do nothing - they need evaluation and possibly remediation just like any other computer, be it PC, mainframe or chip.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Marsh,

As I said earlier, why do scales or a combine harvester need to know what year it is?

I appreciate that farmers are not well-informed of the issues. The manufacturers should have informed them of any major Y2K issues by now (certainly car manufacturers will post to millions of car drivers if there is any defect which could comprise car safety).

I did a search of various farming machine websites but saw no Y2K statements. It looks like it is not a problem to them unless they like going out of business :).

regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Well.

The problems with embedded systems are: (1) The mind-boggling number of different systems out there; (2) The even greater number of different implementations of those systems; (3) The fact that nobody is in a position to generalize from all this diverse potential exposure.

"just another" is correct that the potential exists to code catastrophic failures (with no quick or cheap FOF) into embedded systems. These systems exist, no question about it. But how many there are, and what the impacts may be, and how many have been remediated, is something we can only guess at. Our guesses must be based on test results. The power industry seems to have found very few critical problems after a lot of testing. The level of testing in other industries (like chemical or oil) is an informational black hole. You simply cant generalize that if almost no date bugs were found in the power industry, that this same very low level of incidence will be found in other industries. You gotta test.

Shuggy emphasizes the ease of remediation of a certain class of device. Within a limited scope, this is a Good Thing. But it applies only to (1) fairly new microprocessor-controlled systems with flash ROMs; (2) Where an upgrade has been made available; and (3) Where this upgrade has been performed. And again, nobody has quantified all of this. Also, there's the microcontroller class, and these devices are typically not flashable (at least I've never worked with one that is).

Some (like Paul Davis) point out that engineers design systems with the understanding that they don't last forever, and they break down. The engineering response to this tends to be a combination of making devices easy to replace, overbuilding them for ruggedness, and building redundent (backup) systems. Unfortunately, these techniques are not designed to handle common mode failures.

Most of the evidence we have that these tests are being performed and problems are being addressed is fairly indirect -- we can look at remediation budgets. Apparently GM has remediated their assembly lines. TAVA is a good source of discovered problems. The fact that they've found a lot of problems implies some substantial testing, and the fact that TAVA is also in the business of *fixing* those problems is probably a good sign. We ought to remember also that it's in TAVA's self interest to emphasize the problems they've found, so the listed possible impact is also the worst possible result, however unlikely in normal practice and usage.

The embedded systems arena in general is simply too broad and diverse to make meaningful generalizations. Nothing in this area, from zilch to meltdown, would surprise me.

-- Flint (flintc@mindspring.com), October 28, 1999.

Hi Shuggy,

Now, I understand people's problem when they talk about embedded systems.

You said:

"Bottom line for us is - no Y2K problem. The technology moves so fast that nobody can afford to be left with redundant out-of-date techniques. One of my previous companies started off selling networking products with PROMs in the 1980s which could only be updated by physically removing the chip and putting in the one we sent them. "

You're right, technology moves fast -- in computer-related systems where a device could be obsolete within 6 months.

But, those aren't embedded systems, they're computer-related systems.

Embedded systems are computers built into and an integral part of a non-computer device. Microwave ovens contain embedded systems, "smart valves" contain embedded systems, factory robots contain embedded systems -- but routers, bridges, etc. don't contain embedded systems; they're configurable computers.

The major difference between embedded systems and other kinds of computer systems is that the lifetime of a device containing an embedded system isn't measured in months, it's measured in years and decades. The rate of change is very slow, compared to the computer industry.

Industrial equipment isn't changed just because something new and better is available. These machines are replaced only when maintenance becomes too great or a new machine comes along that will give an order of magnitude increase in production (most companies won't replace their 10, $4 million stamping machines every 6 months).

I have many devices in my home having embedded systems that are over 15 years old (mostly ham radios, and one is over 25 years old). None of them use dates (I hope there are no hidden dates in them), so they'll be okay.

Besides industrial equipment, other equipment with embedded systems are such things as oil pumping and pipeline equipment, shipboard control (and navigation) equipment, electric generation and distribution equipment -- and this type of equipment is generally used until it fails.

Let's just hope few of them fail after 1/1/00.

-- Dean -- from (almost) Duh Moines (dtmiller@midiowa.net), October 28, 1999.

Hi Shuggy,

Re: scales, etc.

Truck scales have just about always had a time/date stamp on them, even 50 years ago. Back then they had a mechanical time clock built into them (I don't recall if the date had to be manually set on a daily basis). This is because they printed out the weight of the truck (and each axle of the truck in many cases) with a time/date stamp -- and the printout was a accounting/legal document that was used to calculate how much a load was worth (or how much to pay the trucker for hauling it). That's just like pre-computer cash registers where the date was set manually (and some had a mechanical time clock).

Newer harvesters are rather expensive pieces of equipment. They go for $1/2 million and up. There are big engines, transmissions and other equipment in them that must be maintained properly so the equipment isn't ruined. That's where the date sensitive embedded's are used -- to make sure the equipment won't run unless maintenance has been performed. (I have one sitting in my yard right now -- it's not mine :) -- I have a 5 acre yard.)

-- Dean -- from (almost) Duh Moines (dtmiller@midiowa.net), October 28, 1999.

Dean,

Thanks for the clarification on truck scales. Sounds like the scales will still continue to weigh even if the wrong year is printed to the accounting form.

As for harvesters, the older ones will still work I presume whilst the newer ones once again are easily remediated via a PROM upgrade? Is there a manual override for these new harvesters and has the manufacturer contacted the owner about upgrades?

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Hello Dean,

Regarding definition of "embedded". I don't know if you are a purist in that sense but the meaning of the word "embedded" has definetely changed and the distinctions are blurring between a computer and an embedded system.

You can now load an entire operating system onto a chip along with a variety of 3rd party products to facilitate the functioning of the machine it resides on. We have moved from the rigid, proprietary configuration to something more dynamic and portable across a wide range of uses. For examples, the embedded operating system VxWorks could run on a PC or a washing machine - take your pick. And of course, Bill Gates is ever-present with his grandiose schemes for Windows CE.

My company touts themselves as an embedded system solutions company and we go to trade shows calling themselves "Embedded Systems Shows" and so on ... and thwese shows are not dominated by the older type of implementations.

But once again the bottom line is that these old and new systems are not catastrophes waiting to happen.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

C'mon Dean, that sounds like a lot of bull to me. Please give us all the make and model of a harvester (or any other farm machinery) that will absolutely not operate unless the "last maintained date" is properly entered? And truck scales do not have embedded chips. The system used to display the output and print out the results may, but the scales do not. The are simple electromechanical devices and the weight of something can easily be found using a simply analog disply with the right calibration.

-- There Is (noone@home.now), October 28, 1999.

Flint,

You are right. There is a mind-boggling number of embedded devices out there but there is also a mind-boggling number of companies and engineers who made them and are capable of fixing them! It is not our job to stand back and do chip or head counts - let the companies get on with it and finish in the time that remains.

I offered the main solutions to embedded code remediation. One can argue about the nuances of particular cases. They will be upgraded, replaced or worked around. The scope for catastrophic multiple failures is in IMHO very small.

The only scope I am worried about is human apathy but that is mainly confined to other continents with varying levels of remediation. I expect the UK to get through on the embedded systems issue okay.

The varying degrees of failure in Asian countries will have an economic impact on us but that is far more preferable to unremediated embedded systems over here.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), October 28, 1999.

Bottom line, gentlemen-- its maybe or maybe not a problem. After cruising through this post plus yesterdays, it sounds like it all depends upon how frequently companies made software/hardware fixes. We all have had customers or worked for companies who were up to date on IT releases, and then there were others who let things slip and had a huge price to pay to get current. Those types will have a bigger price come next year. Would be nice to know which is which, however.

-- Nancy (wellsnl@hotmail.com), October 28, 1999.

Shuggi,

I'm not going to let you get away with this lame and irresponsible debunking attempt. Let's examine your fallacies one by one. the all caps that I use at times are NOT intended to be shouting or anger, but rather as a visual aid to help differentiate at times between my words and yours. ]

You don't have a clue about the Oil or Petrochemical industry problems with embeddeds... so right off the bat you're in trouble when you make glittering generalizations.

---------------

#1. You stated:

"So please tell me how many of these life-or-death chips in stupidly inaccesible areas there are. If no one knows then it is all just specious speculation."

OKAY-- I'LL ACCEPT THAT CHALLENGE! You don't know anything about the oil industry do you?

There's are myriads of embedded systems problems in the oil and petrochemical industry. Here's justone example, and as the article notes -- it could be extremely dangerous:

From the IEA's Oil Industry/Y2K webpage at: http://www.iea.org/ieay2k/homepage.htm

Scroll a third of the way down to this heading and read:

Y2K Readiness in Oil and Gas Industries Mainly US based, some interesting statistics on pipelines and the costs of checking malfunctioning pipelines. "The major problem will be if theres an embedded chip issue in the underwater pipelines, especially the deep ones off the shores of Alaska or in the North Sea where the depth of the sea floor is measured in miles rather than feet. There are only a few vessels capable of taking divers down to those depths, and it would cost around $500,000 a dive. There are two possible scenarios, a chip could either shut down completely, or it could start transmitting erroneous data: for example, about how much fuel is in the pipeline. This would be the more dangerous situation, because the malfunctioning chip would have to be found and replaced. And that could take time."

Now the above is from the following website article:

http://year2000.dci.com/Articles/19981208petrol.htm

SO... this is but one example for the oil industry of which there are tens of thousands. The above applies to a greater or lesser degree to virtually ALL significant offshore oil platforms now in production.

I've got relatives in the refining side who could spend all night listing many different systems in various

units of refineries where life & death safety issues stem from these embedded systems... more on this

later. But lets continue with exposing your ridiculous assertions.

------------

#2... You stated:

" 1. Download the Y2K-fixed image from the provider's web site."

" 2. Receive the new chip by post from the provider."

Again, you've obviously not dealt with the oil industry situations because this is completely off the wall. BTW, Have you ever heard of SCADA? Anyway, the oil industry has embedded systems in the most inaccessable places inside sealed systems. I've got stories from my guys telling me that they've found some sealed systems that must recquire hardware replacement. No longer available and the vendor is no longer in business. The item was a special customized piece for which there are no longer schematics because of prior layoffs inside the oil co itself and no way to trace nor track down any replacement. Then there has been the issue of finding someone to design a new one and have it custom built. Another problem has been certain parts are on backorders of a year or more. They can get it in 2000. But that's a little late...now with the Taiwan quake I'm sure that situation has probably grown worse.

So much for your notions... get into the real world, pal. Or rather, admit that you don't know and can't see the entire picture. None of us do. But some of us have been able to come across enough eyewitness testimonials and related articles from other experts to realize that guys like you don't know the full story and you're only seeing a small slice of the big picture. In your own corner, things may be okay...but this is not true everywhere, and for some, things are indeed extremely critical right now and frankly, in some critical areas, some industries are not getting the job done sufficiently in time.

------------

#3 You Stated:

" 3. What if the provider goes bust? How naive! The free market abhors a profit vacuum, the software is sold to a new company who negotiate a support contract with the buyers to maintain the product, they recruit one or more of the original engineers until the transfer of knowledge is completed. I should know - one of my companies did exactly this for an important customer of a defunct parent company."

Boy, you DOOO live in a vacuum or else you're extremely naive! How ironic that you claim otherwise. I've got a half dozen or more oil embeddeds remediators that would love to get you by the neck and could cite a whole litany of items that are impossible to replace. There's a bunch of Asian chip makers that went bust and are gone and no way to get replacements. PERIOD! In addition, a lot of systems were custom made, ones of a kind. My guys couldn't get replacements, pulled their hair out, thengave up because, lo and behold, management didn't want to stop production to further test other systems on a unit for fear of a permanent fatal shutdown. It is very possible that the oil industry has only testedabout 5% of their embeddeds for compliance issues. There's tens of millions in the oil biz. One offshore rig

may contain 1,000 systems or more. (notice I didn't say chips, I said systems) Many of these are MAJOR systems, big-boys toys.

However, what I shall do is provide you with the URL of one oil equipment supplier -- Baker Hughes Inc... and their Y2K website statements where they list all of their subsidiaries Y2K situations and the compliant status of their products.

BHI

Baker Oil Tools: Baker Process Baker Petrolite Baker Atlas Baker Hughes INTEQ http://www.bakerhughes.com/y2k/mby2k.htm Now, you'll find a lot of their new products are Y2K compliant, a lot of critical systems are NOT yet compliant... you'll also see a lot of obsolescent items that are NOT Y2K. These are indicative of some of the old pieces still out there. EVEN ON COMPLIANT models, you'll find CAVEAT disclaimers for systems that have been placed into more customized applications which means ALL BETS ARE OFF on compliance.

NOW, I've decided to provide just one example for lurkers who are disinclined to go to the links..but here's an example of what I'm talking about...

Centrilift has one critical system that WILL NOT BE MADE COMPLIANT:

Downhole Monitoring Tools & Control Products w/ MicroprocessorsOil Dynamics (ODI) DHS-5000 Down Hole Monitoring System: Non-compliant with Millennium Date Change - Operational Changes Required, Potential Data Integrity Concern. September 9, 1999. Note: Centrilift does not plan to make any product design changes in order to meet Y2K standards.

This disclosure is applicable to the Oil Dynamics (ODI) DHS-5000 Down Hole Monitoring System, (hereinafter referred to as the "Product").

Shuggi-- Note that the date is about 6 weeks ago... and they're not going to even try to meet compliance... AND THEN here's a different twist... from their WESTERN GEOPHYSICAL subsidiary that has its own nightmare of a problem with marine vessels:

The Company has been closely monitoring the previously discovered exposure on two non-compliant field acquisition systems used in the marine vessels. One system impacts 8-9 vessels. The other system impacts 6-8 vessels. The systems are 3rd-party products from a single manufacturer. The Manufacturer states they will make a compliant version of the first of these systems available near the end of June, after experiencing several delays in their scheduled release date. Western will purchase compliant licenses on this system, with expected rollout to vessels by August 1999.

The Manufacturer has advised that they cannot bring the other acquisition system into compliance. As the second system cannot be replaced by year-end, Western is negotiating with the vendor for Western to make necessary modifications to the software internally and test internally to remediate and verify compliance. A priority has been placed on developing contingency plans. Two of these systems are being leased externally and the Company is negotiating an agreement to provide the customer with the internal fixes.

NOW SHUGGI... Keep in mind that this site updates all the time... daily if needed. The above status remains unresolved or it would have been announced. This is a major supplier of embedded systemsto the oil biz...and its got problems. I could go on... for instance... look at the ENRON website and notice how they really take the rest of the oil industry to task for ducking these critical issues. ENRON is at least being candid to a point, but they're the extreme exception.

--------- #4 You stated: "What if the chip is totally inaccessible? Two possibilities:"

OH REALLY... WELL I THINK WE'VE ALREADY COVERED THIS ONE... [I'm not shouting here]

See my above comments regarding offshore oilwells as one example.

a. The date change function is not critical - just disable it remotely or ignore and fix on niggly problems occuring (who cares if the logging function sends screwy dates - tell the PC or mainframe to ignore it if it doesn't already).

=====

Re: point a. -- What a joke? You've got to be kidding, right??? The dates are critical in most of these oil systems. They've got to be there. Some of it is for safety, some of it is for revenue flow documentation for payment to or from a transaction.

=====

b. Catastrophic failure. Having eliminated virtually all chips in the above discussion just how many of these mythical mission-critical chips that no one can get to are out there? Can anyone tell me because I don't know and why put mission-critical chips in such an inaccessible area without some contingency such as a remote command system to access them? That sounds just plain stupid.

SEE MY ORIGINAL #1 point response regarding offshore oil wells. Shuggi, you only show your complete ignorance of not only the oil but the petrochemical and chemical refining industries.

----------------

#5. You stated:

"There is a mind-boggling number of embedded devices out there but there is also a mind-boggling number of companies and engineers who made them and are capable of fixing them! It is not our job to stand back and do chip or head counts - let the companies get on with it and finish in the time that remains."

Yes, in the oil industry, they should have started back in 1990 or 91. They can't begin to get them all

identified and tested, let alone get them all remediated. As one oil engineer in the field told me,

"it's a complete nightmare. There's no way we can even get a majority let alone all of them. We'll be lucky to do 10%, but our PR guys are gonna say we got 'em all, 100%. It's a lie. I know, I was there, I did it and I deliberately missed a bunch because we couldn't get to them anyway," ... and they're old, chances are we'd have permanently taken down the whole unit."

They're telling me that its better to let it run on and risk the blow- out ... at least its more cost effective to the company. What they're doing is "rolling the dice" at a gigantic 'craps table' called Y2K.

----------------

#6 You stated:

"I offered the main solutions to embedded code remediation. One can argue about the nuances of particular cases. They will be upgraded, replaced or worked around. The scope for catastrophic multiple failures is in IMHO very small."

What you've done is showed us that YOU KNOW NOTHING about the oil and petrochemical biz & they're embeddeds problems. There are now various groups... IEE or some other alphabet group, thatI've now lost the link for, as well as allusions by the CIA reports and GAO regarding the oil industry as being in critical shape.

Frankly, I've just touched the tip of the iceberg here. IF you're new here, I've posted some serious reports on the oil industry situation. I don't call it TEOTWAWKI. I don't see it as a 10. For awhile, I'd thought maybe a 5-7, then waffled to maybe 4-7, but I've since adjusted back to and up to 5-8, butmore closely thinking 6, but wouldn't rule out an 8. "10" is impossible-- as that would be planet Earth blows up. 8 would be extremely bad... but I also think its fairly remote, yet. However, you pollies and thegov't and industry actions of late have me wondering if 8 is all that remote anymore. Still, I'm holdingthat 5 or 6 is most likely. Probably severe oil supply/refining disruptions for 90 to 180 days. Minim. of 30 days.

OH... and if the pipelines (SCADA) go down, [and I've heard from a couple of insiders that one major pipeliner is "toast" already and knows it.] then it won't matter how much oil is in the gov't strategic reserves.. which is in salt caves and would need to undergo heavy cleaning before refinement takes place.

This whole oil industry thing is very fragile. WILL IT GO DOWN? We don't know! BUT, the ODDS are a helluva lot higher and frighteningly higher than even most doomers realize. We might have only some minor problems OR it comes staggered with enough ability to keep things rolling... but like someone else here said... the CW is that it will all blow (if it does) within a 1 week window if its gonna blow.

THEN OF COURSE, we didn't discuss the foreign oil problems...which are even worse ... just ask the CIA about that or see the IMM report.

-- R.C. (racambab@mailcity.com), October 29, 1999.

Lars,

You stated: "What has always seemed odd to me is that there is relatively little alarm (of which I am aware) about the embeddeds in the engineering world. If we were so hugely vulnerable here then no combination of lawyers and marketers could cover this up. (and why would they want to?)."

My response: Oil engineers are well aware and are very afraid to speak out. Corporate honcho's have decided that the only recourse financially is to roll the dice, because they were all asleep at the switch 8-9 years ago when they should have been doing something about all of this. My sources within the oil industry are terrified to speak out. Therefore, the public is left in the dark.

-- R.C. (racambab@mailcity.com), October 29, 1999.

RC:

You quoted the following -

Y2K Readiness in Oil and Gas Industries Mainly US based, some interesting statistics on pipelines and the costs of checking malfunctioning pipelines. "The major problem will be if theres an embedded chip issue in the underwater pipelines, especially the deep ones off the shores of Alaska or in the North Sea where the depth of the sea floor is measured in miles rather than feet. There are only a few vessels capable of taking divers down to those depths, and it would cost around $500,000 a dive. There are two possible scenarios, a chip could either shut down completely, or it could start transmitting erroneous data: for example, about how much fuel is in the pipeline. This would be the more dangerous situation, because the malfunctioning chip would have to be found and replaced. And that could take time."

I am still a little confused about the "hard to reach" chips thing. How do the companies that use such a chip budget for maintenance costs? Why put a relatively inexpensive chip into such a difficult and expensive to reach place ($500,000)? What happens when a chips fails for a non-Y2K reason? Or do these types of chips never fail?

-- Johnny Canuck (j_canuck@hotmail.com), October 29, 1999.

Its very simple. Go to [url] www.iee.org/2000risk/Casebook/eg_index.htm[/url] and go all the way down to "petrochemical". That link says it all.

-- It (Says@It.All), October 29, 1999.

I went to the URL you mentioned and didn't find a description of a "hard to find" chip problem.

In the problem mentioned "EG-07" what would the plans be to deal with a non-Y2K induced failure?

EXAMPLE NO EG-07

Equipment Type DCS Industry Sector Oil & Gas

PC or Computer based No System Age 6 years

Application DCS control system control for petrochemical plant

Description of the Problem Online rollover to Year 2000

How was it Identified During testing. Offsite testing on a testbed was performed with satisfactory results. Upon testing of stations on site, control was no longer possible after the system had rolled over to Year 2000. It was not until this problem was evident on three of the four operating stations was testing aborted.

What was the Solution No known workaround. Plant had to be operated from one station until problem could be rectified

Consequences for the SYSTEM System Stops

Consequences of failure to the BUSINESS Near catastrophic. Limited reliability and operability of plant. Reduced production

OK - the above is copied directly from the web site. I'm still confused though; what do they do when they have

-- Johnny Canuck (j_canuck@hotmail.com), October 29, 1999.

Johnny,

They send down diving teams in the shallow areas and for the deep stuff they'll send down a sub unit to make repairs...very expensive and I'm sure the bean counters take that into consideration when making budgets.

For the most part, those systems are relatively trouble free. Those rigs generally are "high producing" wells that provide mega bucks. So, it's still a financially viable solution to send down an expensive sub operation. Consider the expense of drilling the hole and putting up the platform in the first place. The real problem is the cost in terms of production downtime.

-- R.C. (racambab@mailcity.com), October 29, 1999.

Shuggy --

Thanks for the post. Some good information here. I feel better having read it. Your experience appears to be more up-to-date than mine in the area of embedded.

A couple of points though. This is *one* industry. I am glad it is compliant, but I don't necessarily know that the techniques translate directly to other industries. (See also what Flint wrote). And the other is that it still doesn't address some of the older stuff out there.

Still, on the whole, a good post, and encouraging information.

-- just another (another@engineer.com), October 29, 1999.