Hi All,

Perhaps most people missed this item from last summer. Here's a message I posted on c.s.y2k regarding an embedded controls conference. Notice that the compliance failures are well above the 1.5% or 6% being thrown around here.

General industry assumptions put the expected failure rate somewhere around 15% -- so 6% is low. (I hope it formats all right.) ------------------------------------ You can find the full article at: http://www.controlmagazine.com/0398/c0200398.html#1

================================= Dave Woll, vice president for services, ARC, began the session by offering some data. He said of the 3.3 billion micro-controllers embedded in the automation infrastructure, 50 million will have Y2K anomalies. "Many of them are older ones that no one knows the code for," he said. "Your company may have bought lots of custom code in the '70s and '80s. Now it's yours and no one else's."

As a reference point, Woll reviewed the Dept. of Defense Year 2000 project inventory report. He said of 3,962 applicable systems, 582 were OK, 623 were being renovated, 628 were retired, and the balance of 1,900 was being assessed. The numbers suggested that about 25% of all the systems would require some level of fixing.

"If you're in a position to speak up, make sure company officials recognize the liability they carry as officers of your company," warned Woll. "Y2K losses will not be considered an act of God. There are serious legal, risk, and insurance implications."

Patrick Meehan, Y2K program manager, DuPont Operations, presented the large-user perspective. "Let's face it, there's not much upside and a lot of downside," he offered. He sees that 50% of DuPont's work will be with process control devices and systems and his current estimate is that, while 100% will be examined, 10-15% will need remediation.

Several presenters mentioned the need for examining both device and system, saying just because the device is OK doesn't mean the system is. Distributed systems with layered, embedded applications can make the search for every relevant date and time function like looking for needles in haystacks. Meehan recalled that some of the assessment and trial ordeals reminded the participants of a "plug and pray" analogy.

Newell provided some sobering examples during his Honeywell presentation. He spoke of a recent CAP Gemini Poll that showed five of every six Fortune 500s haven't begun the fix phase. "One in four has started the preliminary phase but only one in six has a final plan," he reported.

************* NOTE THIS !!

Richard Ryan, president, Rockwell Software, stood in for his Rockwell hardware cousins and reported that a number of semiconductor plants-operating with a lot of home-grown devices-found two in every three devices had a compliance problem. He added that a Lucent [Technologies] factory reported four of every five failed, and some Rockwell plants had 100% compliance failure in the devices tested.

*************

Ken Owen, TAVA Technologies, Englewood, Colo., invoked the conference's only Titanic reference by suggesting that companies have tended to look only at the tip of the iceberg. "The damage to the systems that help you count your money are getting a lot of attention, but the 90% underwater-the controls -is the structure that makes the money, and it's not getting enough attention."

His company's research concluded that problems will vary. "It will range from simply having erratic production problems, to having lines that don't run because the manufacturing systems have stopped, to having serious safety and quality problems because the manufacturing systems make bad decisions," he asserted.

Sounding a familiar alarm, he concluded by saying, "Do the math. If you started January 1, 1998, you spent one month negotiating what you think you need and you'll need two months to test a pilot program. After that, you'll need five months to conduct a full inventory and assessment of what you have and another five months planning the steps to convert what needs to be converted. Our work so far indicates you'll need ten or more months to remediate, and then-sorry-time's up." ================================

The NOTE above points out that while embedded systems as a whole may have a 5% failure rate for year-related problems, and non-essential systems (the Quantas catering example) may be only 3% -- the critical infrastructure manufacturing is reporting:

66% (semiconductor), 80% (Lucent Technologies -- ex-AT&T manufacturing arm) and 100% (Rockwell)

system failures. Also, the DoD report implies 25% of it's systems need fixing.

You also may recall the Fortune article about embedded systems where the CIO of GM called his company's situation with these systems 'catastrophic.'

-- Dean Miller, CDP -- from Des Moines ---------------------------------------

-- Dean -- from (almost) Duh Moines (dtmiller@nevia.net), March 25, 1999

Answers

Hi Dean!

Thought this might fit well with the above post.

Brian

Finance - LU form 10-Q =
http://sec.yahoo.com/e/l/l/lu.html

---

February 16, 1999

LUCENT TECHNOLOGIES INC (LU)
Quarterly Report (SEC form 10-Q)

MANAGEMENT'S DISCUSSION AND ANALYSIS OF RESULTS OF OPERATIONS AND
FINANCIAL CONDITION

Readiness for Year 2000: Lucent is engaged in a major effort to minimize the impact of the Year 2000 date change on Lucent's products, information technology systems, facilities and production infrastructure. Lucent has targeted June 30, 1999 for completion of these efforts.

The Year 2000 challenge is a priority within Lucent at every level of Lucent. Primary Year 2000 preparedness responsibility rests with program offices which have been established within each of Lucent's product groups and corporate centers. A corporate-wide Lucent Year 2000 Program Office ("LYPO") monitors and reports on the progress of these offices. Each program office has a core of full-time individuals augmented by a much larger group who have been assigned specific Year 2000 responsibilities in addition to their regular assignments. Further, Lucent has engaged third parties to assist in its readiness efforts in certain cases. LYPO has established a methodology to measure, track and report Year 2000 readiness status consisting of five steps: inventory; assessment; remediation; testing and deployment.

Lucent is completing programs to make its new commercially available products Year 2000 ready and has developed evolution strategies for customers who own non-Year 2000 ready Lucent products. The majority of the upgrades and new products needed to support customer migration are already generally available. By the end of 1998, all but a few of these products were generally available.

Lucent has launched extensive efforts to alert customers who have non-Year 2000 ready products, including direct mailings, phone contacts and participation in user and industry groups. Lucent also has a Year 2000 website www.lucent.com/y2k that provides Year 2000 product information. Lucent continues to cooperate in the Year 2000 information sharing efforts of the Federal Communications Commission and other governmental bodies.

Lucent believes it has sufficient resources to provide timely support to its customers that require product migrations or upgrades. However, because this effort is heavily dependent on customer cooperation, Lucent continues to monitor customer response and will take steps to improve customer responsiveness, as necessary. Also, Lucent is continuing contingency planning to address potential spikes in demand for customer support resulting from the Year 2000 date change. These plans are targeted for completion by April 30, 1999.

23 Form 10-Q - Part I

          &
nbsp;    MANAGEMENT'S DISCUSSION AND ANALYSIS OF RESULTS 
OF
           &
nbsp;           
OPERATIONS AND FINANCIAL CONDITION

Lucent has largely completed the inventory and assessment phases of the program with respect to its factories, information systems, and facilities. Approximately, two-thirds of the production elements included in the factory inventory were found to be Year 2000 ready. The factories have commenced the remediation phase of their effort through a combination of product upgrades and replacement. Plans have been developed to facilitate the completion of this work, as well as the related testing and deployment, by June 30, 1999.

Currently, approximately 65% of Lucent's information technology infrastructure has been determined to be Year 2000 ready and is deployed for use. Approximately, 65% of our business applications lines of code that are supported by Lucent's information technology group are now Year 2000 ready and have been deployed or are awaiting deployment. LYPO is monitoring the progress of readiness efforts across Lucent, with a special emphasis on the early identification of any areas where progress to-date could indicate difficulty in meeting Lucent's June 1999 internal readiness target date. Lucent is developing specific contingency plans, as appropriate.

Lucent is also assessing the Year 2000 readiness of the large number of facilities that it owns or leases world-wide. Priority is being placed on Lucent-owned facilities, leased facilities that Lucent manages and other critical facilities that house large numbers of employees or significant operations. Based on the results of these assessment activities, Lucent plans to complete remediation efforts by March 31, 1999 and complete development of applicable contingency plans by May 31, 1999.

To ensure the continued delivery of third party products and services, Lucent's procurement organization has analyzed Lucent's supplier base and has sent surveys to approximately 5,000 suppliers. Follow-up efforts have commenced to obtain feedback from critical suppliers. To supplement this effort, Lucent is conducting readiness reviews of the Year 2000 status of the suppliers ranked as most critical based on the nature of their relationship with Lucent, the product/service provided and/or the content of their survey responses. Almost all of Lucent's suppliers are still deeply engaged in executing their Year 2000 readiness efforts and, as a result, Lucent cannot, at this time, fully evaluate the Year 2000 risks to its supply chain. Lucent will continue to monitor the Year 2000 status of its suppliers to minimize this risk and will develop appropriate contingent responses as the risks become clearer.

The risk to Lucent resulting from the failure of third parties in the public and private sector to attain Year 2000 readiness is the same as other firms in Lucent's industry or other business enterprises generally. The following are representative of the types of risks that could result in the event of one or more major failures of Lucent's information systems, factories or facilities to be Year 2000 ready, or similar major failures by one or more major third party suppliers to Lucent: (1) information systemscould include interruptions or disruptions of business and transaction processing such as customer billing, payroll, accounts payable and other operating and information processes, until systems can be remedied or replaced; (2) factories and facilitiescould include interruptions or disruptions of manufacturing processes and facilities with delays in delivery of products, until non-compliant conditions or components can be remedied or replaced; and (3) major suppliers to Lucentcould include interruptions or disruptions of the supply of raw materials, supplies and Year 2000 ready components which could cause interruptions or disruptions of manufacturing and delays in delivery of products, until the third party supplier remedied the problem or contingency measures were implemented. Risks of major failures of Lucent's principal products could include adverse functional impacts experienced by customers, the costs and resources for Lucent to remedy problems or replace products where Lucent is obligated or undertakes to take such action, and delays in delivery of new products.

24 Form 10-Q - Part I

          &
nbsp;    MANAGEMENT'S DISCUSSION AND ANALYSIS OF RESULTS 
OF
           &
nbsp;           
OPERATIONS AND FINANCIAL CONDITION

Lucent believes it is taking the necessary steps to resolve Year 2000 issues; however, given the possible consequences of failure to resolve significant Year 2000 issues, there can be no assurance that any one or more such failures would not have a material adverse effect on Lucent. Lucent estimates that the costs of efforts to prepare for Year 2000 from calendar year 1997 through 2000 is about $535 million, of which an estimated $280 million has been spent as of December 31, 1998. Lucent has been able to reprioritize work projects to largely address Year 2000 readiness needs within its existing organizations. As a result, most of these costs represent costs that would have been incurred in any event. These amounts cover costs of the Year 2000 readiness work for inventory, assessment, remediation, testing and deployment including fees and charges of contractors for outsourced work and consultant fees. Costs for previously contemplated updates and replacements of Lucent's internal systems and information systems infrastructure have been excluded without attempting to establish whether the timing of non-Year 2000 replacement or upgrading was accelerated.

While the Year 2000 cost estimates above include additional costs, Lucent believes, based on available information, that it will be able to manage its total Year 2000 transition without any material adverse effect on its business operations, products or financial prospects.

The actual outcomes and results could be affected by Future Factors including, but not limited to, the continued availability of skilled personnel, cost control, the ability to locate and remediate software code problems, critical suppliers and subcontractors meeting their commitments to be Year 2000 ready and provide Year 2000 ready products, and timely actions by customers.

-- Brian (imager@ampsc.com), March 26, 1999.

Unless my calculator has gone belly up, 50 million divided by 3300 million gives 1.5151... per cent.

And you need to elucidate just what you mean by Rockwell having a 100% failure rate. I am familiar with quite a bit of Rockwell equipment, and it does not have a failure rate of even 1%.

-- Paul Davis (davisp1953@yahoo.com), March 26, 1999.

Hi Brian, Paul,

Brian, thanks for the info. Lucent's latest 10-Q appears to say that 'only' 33% of it's production lines have problems (with one factory apparently at 80%).

Paul, you'll have to take up the Rockwell numbers with (pasted from the conference report) "Richard Ryan, president, Rockwell Software."

I imagine that Rockwell found their plants had high rates of failures, while the PLC's and other systems they produce are compliant -- at least the currently supports systems. The low rate of failures seems to be true of A-B, Foxboro, Rosemount-Fisher, etc. current equipment.

Also, the comment about the 3.3 billion vs. 50 million is talking chips, not systems. We have to keep our definitions solid or we'll never get anywhere in discussions of what the actual conditions are in the field.

Many, many system boxes use multiple microcontrollers, PLC's and the like. Especially the older systems where the controllers had clock rates around 1 MHz. And there could be 10's to 1000's of these system boxes/PLCs in one embedded system.

All it takes to bring down a system like this (invalidate the process or make the system unreliable) is one of the microcontrollers in one of the boxes producing bad data. Then usually the entire system shuts down to prevent damage to the controlled machines. (Do you want your printer to keep feeding paper if it thinks there's a paper jam?)

Oh, the 50 million number is probably quite low. I recall reading reports of one power company saying they have over a million embedded controllers or smart sensors installed with an non-compliance rate of around 1% (10,000). More complex production systems have larger numbers of controllers and certainly could have higher percentages of non-compliant, date-sensitive, controllers -- as evidenced by the reports from the conference I posted.

BTW, somewhere in the archives of c.s.y2k are comments from a Kraft foods exec where he said that several of Kraft's production plants (and even food lines) were being discontinued because of Y2K problems.

-- Dean -- from (almost) Duh Moines (dtmiller@nevia.net), March 26, 1999.

Lucent and Rockwell have problems with the suppliers of the production line equipment. I can think of one specific equipment supplier to both firms, that is on its way from "Lucent Quality Partner" to "Lucent Lawsuit Target".

Maybe getting laid-off by that firm was a good deal after all. I don't think being part of the obvious "fire drill" would be too fun. They've gotten the hardware upgrades for their assembly machines in hand. But the software change to "Y2K compliant NT4.0" seems to be running into snags with the "Y2K patch of the month club" delaying their software release.

This is the kind of embedded systems problems that are going to result from Y2K. It's going to put some companies out of business if they don't catch it in time. And it's certainly going to wreck the suppliers no matter how Y2K turns out.

Look for a lot more of these kinds of stories to come out as the year progresses. The firm I know of has at least two thousand problem assembly systems sold round the world. That's two thousand failures, each of which can stop someone's assembly line and two thousand lawsuits to boot.

WW

-- Wildweasel (vtmldm@epix.net), March 26, 1999.