READ ONLY - Y2K List of Failures Part 1 - Please do not post to this thread

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

1. Source: National Radio Astronomy Observatory - a facility of the National Science Foundation

http://info.cv.nrao.edu/y2k/sighting.htm

(Excerpts that were failures in 1999 so far)

1999 January 25: Still some problems with 00 credit cards. Information Week reports that Visa still receives 100 to 150 reports a month that credit cards have been declined by point-of-sale systems because a card's expiration date is in the year 2000. MasterCard says it sees a handful of such incidents a month, usually at small, single-store operations. (The report does not compare this to the number of transactions now being made with '00' cards, which were embargoed for a while when point-of-sale problems first surfaced in 1997; the fact that the failure rate is now this small may be one of the Y2K success stories!).

1999 January 10: NY auto sticker shock. According to an AP report in the Fort Worth Star-Telegram Environmental Systems Products in Bohemia, NY, one of three companies that supplies automobile emission-testing equipment for use in New York state discovered that older equipment it had been phasing out could not produce a 00 windshield sticker for drivers whose vehicles had passed inspection late in 1998. '00' stickers were needed starting 1998 December 22 because New York gives an 8-day grace period for one-year validations. The older Environmental Systems equipment instead issued stickers saying '91'; many inspection stations put these on car windows, and parking agents began ticketing the apparently out-of-date cars. Environmental Systems reported the problem to the state/. Procedures were set up to quash the tickets, and a crash effort was maded to complete the transition to newer equipment. "It cost a lot of money," said James Richardson, the company's director of operations.

1999 January 6: Australian hospitals concerned. The Australian reports that almost a third of computer-related equipment in several South Australian hospitals, including cardiac monitors and drug distribution systems, have failed Y2K testing. "Of the 22,761 systems tested in South Australia, 7042 were non-compliant." Bob Hancock, the 2000 co-ordinator for Royal Adelaide Hospital, stated that the hospital would have to replace its monitoring and pharmacy systems because of non-compliance. (This report is a marginal "sighting" because it is not explicit about the nature of the non-compliance, or its potential consequences, but nevertheless implies that it is "serious".)

1999 January 5: Embedded chip locks out workers in Y2K-1 report. According to a UPI report, David Sterling, the head of Sterling and Sterling Insurance of Long Island, NY, states that a flaw in the hardware of his brand new card-access security system locked over half of his staff of 85 workers out of their office building on 1999 January 4 because the year had changed from 1998 to 1999. Sterling says he had been skeptical about predictions of malfunctions in date-sensitive chips, but now has a new attitude. "What scared me was the problem was not with the personal computer, which we checked. It was a chip in a control panel."

1999 January 4: Y2K arrives early for small WV firm. Business Week reports how the look-ahead time in accounting software made Lin Electric in Bluefield, WV, hit Y2K trouble sooner than expected. When the small company, which specializes in reconditioning electric motors, closed its books on fiscal 1998 and tried to open fiscal 1999, not only did their computer crash but the accounting system from Cougar Mountain Software Inc. locked up so tight it couldn't be restarted. Lin Electric had been advised of the problem but had intended to tackle it early in 1999, not realizing when it would strike. The Y2K bug can bite accounting packages early because they must work with periods long enough to handle overlaps between corporate fiscal years and calendar-year tax periods. Thus, when Lin Electric opened fiscal 1999, the software tried to create monthly entries extending into mid-2000. When this process got to 2000 January, it calculated the two-digit year "00" as 1944. "It has to do with the computer's BIOS [basic in/out system] chip and how it calculates dates," stated David Lakhani, sales manager of Cougar Mountain Software in Boise, Idaho. Lin Electric, a company with just 20 employees, processed its payroll checks by hand while the problem was being fixed.

1999 January 3: British examples. According to the Sunday Times, the bug has already affected numerous organizations. Examples: An oil company found that a dockside crane would not operate because an embedded chip thought an inspection was overdue; a National Health Service trust found that its computers would not process patient appointments beyond 1999 December; Halifax Building Society sent letters to customers informing them of new policies that were valid from 1999 to 1900.

1999 January 1: Taxis bugged by Y2K-1 in Singapore and Sweden.According to an Associated Press report, allegedly Y2K-compliant taxi meters malfunctioned today in Singapore and in Sweden. Some in Singapore stopped working altogether, while those in Sweden calculated fares incorrectly.

1999 January 1: Y2K-1 in Sweden. Police units at Stockholm's Arlanda airport, and at Gothenburg and Malmo airports got a foretaste of the Y2K bug when their computers malfunctioned at midnight on 1998 December 31. The bug hit airport police offices that issue immediate, temporary passports to last-minute or forgetful travellers. In another Swedish example, customers of Norway's state oil company Statoil, which operates about 600 gas stations in Sweden, could not use their credit cards because pumps had been programmed to accept them only through 1998 December. "There was nothing wrong in the data technology, but rather it was we who programmed badly," said Statoil spokesman Henrik Siden. These incidents appear to have been caused by the use of 99 as an end-of-file code.

2. Source: Mardon Century Experts, Incorporated

http://www.mardon-y2k.com/99newpage.htm

(This site has 1999 reported problems by the these industry sectors: Law Firms, Distribution, Grocery/Food Services, Insurance Associations/Mailers, PCs Personal & Business, Consulting, Remediation Projects' Impacts, Education (Higher Ed) Entertainment, Law Enforcement, Manufacturing, Transportation, Public Utilities, Media, Federal, Education (K-12), Public Sector, and Health Care. Some sectors, as of 1/27/1999 had no reported problems: like public utilities and entertainment.)

Federal Industry Sector - updated 1/22/1999:

As expected, government computers in Washington and in several states were unable to sign people up for unemployment benefits using standard forms, which set a date one year ahead as the end of the benefits. But, as previously planned, those agencies continued to sign people up for the benefits last week by plugging in Dec. 31, 1999, as the ending dates. John Koskinen, chairman of President Clinton's Council on Year 2000 Conversion, said the unemployment benefit scramble underscored the need to plan for breakdowns well in advance. He said he was worried that last week's calm might encourage local governments and small businesses to settle on "wait until it breaks" policies. Source: The New York Times on the Web

Education (K-12) Sector - updated 1/13/1999:

Type of failure: Incorrect calculations caused by adding 1 to current year and comparing to 1998. Impact of failure: Batch payroll job for supporting services employees received incorrect calculation results. Defect detected during the normally proofing and verification of check registers by the payroll staff. Additionally a related on-line ADSO program abnormally terminated due to the same defect. No resolution as of this report. Source: Mardon Direct Reporting Project

Public Sector - updated 1/13/1999:

County Government - Permits Issuance Type of failure: Incorrect output due to date of issuance being 99 and end of permit being 2000 (showing as 1900) - Impact of failure: Permits in abeyance. No resolution as of this report. Source: Mardon Direct Reporting Project

Works Compensation - Type of failure: Software calculating 99+1 = 00 and making it 1900. Impact of failure: A bookkeeper in California and received the monthly form from the State Workers Compensation Board to complete, pay and return. This one had all the rates for employees wrong. Upon contacting the agent, the bookkeeper was told "you've been bitten!"...their computer is reading the insurance programs 1999-2000 only it read it 1999-1900. He said thousands went out and they are being bombarded with calls. Simple enough to fix, just put in the figures the form recipients know are right. But State Comp still has to figure it out when the checks come in. Source: Mardon Direct Reporting Project

City Government - Inventory - Type of failure: Software: a date validation check stating "If year is less than 88 or equal to or greater than 99 then year is invalid". Impact of failure: A Canadian city government suffered a "99" failure on Jan 4, 1999. In the Inventory System, when an item is returned to stock, a date of return must be entered. It would not accept '99'. The problem was found, reported and corrected in less than 1 hour. Source: Year2000.com Maillist

State Government - Budget and Accounting - Type of failure: Software: The problem is with a date range of next business date plus or minus 1 year. Because the date is in MMDDYY format , the year of the incoming record must be > 98, but < 00. So all 99 dates are kicking out. Impact of failure: Every record on the tape created by Budget and input into Accounting is rejecting beginning with the tape created on 1/5/99, the first run of 1999. This problem was overcome by inserting the segments online rather than using the batch job as a short-term solution. The volume of rejected records is low. Source: Year2000.com Maillist

Health Care - updated 1/20/1999:

Medical Devices - Type of failure:Defective embedded microprocessor and related software. Impact of failure: Incorrect time/date on printed medical records that might lead to erroneous filing. Incorrect operation caused by rollover to 1999. Source: FDA Medical Alert

Consumer - Prescriptions/Insurance - Type of failure: Software or human error. Impact of failure: Pharmacy Systems in the New York and Washington areas indicated people insured by PCS Health had expired insurance. Approximately 50,000 insured individuals were impacted. Many were required to pay the full cash for their prescriptions. While company officials were quick to note that this was NOT a Y2K problem, Mardon will leave this entry open until such time as the companies involved disclose what the cause of the failure was. Source: Washington Post

Patient Care - Medical Devices Type of failure: Embedded Systems within an IV drip unit Shut down at 98-99 rollover. Impact of failure: Personnel had been warned to watch for problems and detected problem shortly after they shut down. No patients' health was compromised by failure. Note: The vendor had indicated that the monitor was OK for Y2K. Source: Mardon Direct Reporting Project

Swiss Hospitals - Admitting System - Type of failure: Software: System programmed to compare input data with the date a year in the future. This was programmed in 6 digits as "01.01.00". The system interpreted this as 1 Jan 1900. Impact of failure: The hospitals in the canton Waadt spent the 1st and 2nd of January 1999 fighting with the computer problems that are expected for the year 2000. Except for the University Hospital in Lausanne, the computer systems for admitting patients in all of the hospitals of the canton were down for 36 hours. Specialists were able to fix the problem, according to a spokeswomen for the hospitals in the newspaper "24 Heures" (24 hours) on Wednesday. Source: Year-2000 computer problem in Waadtlaender hospitals Lausanne

Ultrasound Equipment Users - Type of failure: Embedded system defect. Impact of failure: This appears to be that the initialization routine for startup after replacement of a dead battery has some logic in it that cannot operate with a "00" date. Impact of failure: General Electric Medical Systems has noted on their Web Site that a version of their ultrasound units (the RT 5000, RT 6000 and RT6800) have Year 2000 impacts. If the lithium battery fails during 2000, it will be necessary to replace the CPU board. A battery failure at any other time will not require a CPU board replacement. Source:Year2000.com mail list

Multiple Medical Devices - Type of failure: Embedded Systems & software defects. Impact of failure: An Australian group of medical centers tested various devices and found many fail in either 2000 rollover, holding the 2000 date, or recognizing 2000 is a leap year. The list currently contains 51 devices Source: Mardon Direct Reporting Project

Manufacturing Industry - updated 01/13/1999:

Type of failure: Out of date Accounting Software. Impact of failure: Not only did the electric motor rebuilding company's computer crash but also the accounting system locked up so tight it couldn't be restarted. Payroll checks for the 20 employees at Lin Electric, which specializes in reconditioning electric motors, had to be written by hand. "We're a small company, so that wasn't a big deal," says Taylor. "Our people didn't suffer during the holidays." Source: Mardon Direct Reporting Project

Insurance Industry - updated 01/20/1999:

Type of failure: Microprocessor In Control Panel. Impact of failure: On Monday, workers tried to get into their office building using the new card-access security system, which would not allow them in because it failed due to a '99' rollover bug. Source: UPI Release Via http://y2kreview.com/990106.htm)

Type of failure: Software. Impact of failure: A 4th quarter Field Bulletin announced to the field agents that the company had achieved "full Y2K compliance" and that customers should be informed of this stupendous feat whenever they asked. Last week, however, the agents received another Field Bulletin which stated that an error was discovered in a program that calculates New Home Policies (aka "Remediation"). It seems that data calculations in which the year 1999 is entered, the data are calculated from 1899 instead. Source: alt.talk.year2000

Home and Business PC Users Industry - updated 01/18/1999

Type of failure: Software. Impact of failure: Intuit's Quicken'99 fails with a "divide by zero" message when a transaction dated in January 1999 is recorded in the Auto category and its "Home and Car Center" is opened. Source: http://www.intuit.com/support/quicken/faqs/win2/1913.html

(Check out the link below. It covers "various Y2K readiness projects that have caused external impacts to providers, clients, and/or customers." In other words, the process of remediation itself has sometimes caused failures, and this is a list of some - including what happened recently in the U.S. Senate) http://www.mardon-y2k.com/projbad.htm

3. Source: Electric Power Research Institute Proceedings from EPRI Embedded Systems Workshop, Proceedings dated 10/4/1997

http://www.euy2k.com/reallife.htm

(Excerpt)

Y2K testing was conducted on a generator temperature control system at a power plant in the United Kingdom. To test for Y2K compliance, the control system clock was set to just prior to midnight, Dec. 31, 1999. Twenty seconds past midnight, the unit tripped on high generator temperature.

It turns out the process value for the control valve for generator cooling is integrated over time for smoothing and when the time moved past midnight from '99 to '00, the PV was integrated over infinity. The valve closed (fail safe), tripping the unit on high generator temperature. If this were an isolated incident, the industry would evolve through the year 2000 with little difficulty. However, the algorithms used in this control system are common throughout Europe and most systems are vulnerable to the problem. Loss of numerous generating units simultaneously in the United Kingdom could be devastating to the country.

4. Source: The Cassandra Project Web Site

http://millennia-bcs.com/examples.htm

(Excerpts from the "Year 2000 Problem Occurrences" section)

When Chrysler Corp. shut down its Sterling Heights Assembly Plant last year and turned all the plant's clocks to Dec. 31, 1999, executives were expecting to find computer glitches associated with the date change from 1999 to 2000. But they weren't expecting quite so many glitches. "We got lots of surprises," said Chrysler Chairman Robert Eaton. "Nobody could get out of the plant. The security system absolutely shut down and wouldn't let anybody in or out. And you obviously couldn't have paid people, because the time-clock systems didn't work." Ottawa Citizen, 5/15/98

This past April, the computer network that schedules patient appointments at three hospitals and 75 clinics in Pennsylvania shut down -- all because one person punched in an appointment for January 2000. As reported in Money Magazine

Royal Sutherland Hospital in New South Wales has identified two processors, a laser camera and an ultrasound machine which will be rendered inoperable come 2000. The value of the equipment is estimated at more than $550,000 and it must be replaced. [Reported in Australian Financial Review]

At midnight on Jan. 1, 1997, 660 process control computers that run the smelter potlines at the Tawai Point Aluminum smelter in Southland, New Zealand, could not account for an extra day stemming from the 1996 leap year and crashed. Five pot cells were ruined, leaving the aluminum company with a repair bill estimated at more than $570,000. Two hours later, Comalco's Bell Bay smelter in Tasmania shut down with the same problem, according to reports in the New Zealand Business Herald and The Dominion, in Wellington, New Zealand.

On Jan. 1, 1997, the millennium bug hit a law enforcement computer in New Zealand. The system, which controls criminal records, driver's licenses, vehicle registration, and more, wouldn't let police set court dates two years hence. Files had to be processed manually, and the system will be replaced by early 1999, according to reports in the New Zealand Business Herald.

When the Hawaiian Electric utility in Honolulu ran tests on its system to see if it would be affected by the Y2K Bug, "basically, it just stopped working," says systems analyst, Wendell Ito. If the problem had gone unaddressed, not only would some customers have potentially lost power, but others could have got their juice at a higher frequency, in which case, "the clocks would go faster, and some things could blow up," explains Ito.

In 1993, the Associated Press reported that Mary Bandar, a 104-year old resident of Winona, Minn., turned down an invitation to attend kindergarten. A computer triggered by the fact that she was born in 1888, fired off a notice to begin school in the fall.

Air traffic controllers at an emergency meeting of the International Federation of Airline Controllers (January 1998) simulated the year 2000 date change. Their screens went blank.

5. Source: The Journal of Commerce online

http://www.joc.com/issues/990119/i1nsur/e23991.htm

(Excerpts)

With 2000 nearly upon us, a survey of 110 U.S. corporations, 12 federal, state and local government agencies, and 12 industrial sectors found that a large number of the respondents have experienced Year 2000-related failures, and nearly all of them -- 98% -- expect more such failures in 1999. The quarterly survey was conducted by Cap Gemini America, a Y2K consulting and services provider. The report found that every company is now developing contingency plans to prepare for expected failures. In a similar study done in April, only 3% were putting together such plans. The number of companies that are very likely, or potentially likely, to stop doing business with noncompliant suppliers rose to 69%, up from 60% in the third quarter. The percentage of firms planning to incorporate Year 2000 compliance into marketing messages increased to 55% from 50% in October.

The study found that the percentage of companies with a process in place to validate renovated code prior to testing rose substantially in the past quarter from 16% to 62%. More than half of the respondents -- 55% -- have already experienced a Year 2000-related computer error. This figure is up from 44% in October and 40% in July, according to Cap Gemini. Three quarters of the organizations -- 74% -- expect that more than half of their systems will be tested and compliant by Jan. 1, 1999, a percentage unchanged since October, according to the survey. However, 92% of the companies report missing Year 2000 plan deadlines more frequently, up from 84% last July. In regards to Y2K budgets, companies continue to underestimate their Y2K spending. Eighty-four percent classified their cost estimates as "too low," compared with 82% in the third quarter and 87% in the second quarter. The quarterly survey is conducted by Rubin Systems.

6. Source: Peter De Jager - The Year 2000 Information Center

http://www.year2000.com/y2kbugbytes.html

(As of 1/27/1999, here is what was in the "Current Articles" section for January, 1999 - I include this here since if you are interested in any particular article you can go to the Site and use the hotlinks there. Most of the titles below describe the type of failure. Since then, the site has been updated to include February and March failures also)

Information Week: Y2K Glitches Surface -- Report Shows Diverse Problems

One-third of ITAA respondents report Y2K-related problems

Information Week: Millenium Bug Bites Retailers Credit cards

Boston Globe: Prodigy to shut down online service over Year 2000 problem Prodigy "Classic" termination affects 208,000 subscribers

Minneapolis Star Tribune: 2000 bug a reason for store's closure Grocery store closes doors

Westergaard Year 2000: Evolution of a Myth Vehicle inspection stickers

ZDNet News: Some Y2K bugs strike early 105-year-old woman invited to enroll in kindergarten

Edmonton Journal: Millennium bug strikes U of A early Phone system

TechWeb: Computer Flaw Triggers Early Data Release Early release of producer prize index (PPI)

Roll Call: Senate Isn't Paying Bills on Time - Rent Checks Running Weeks Behind Schedule The Senate

ZDNet News: Computer Date Glitch Locks Workers Out Security system

Business Week Online: Still a year to tackle the Y2K bug? Think again Accounting software

South China Morning Post: Bug puts byte on harbour master's chips Vital information system tracking vessels in and out of Hong Kong

Tennessean : Y2K bug biting before 2000 External defibrillator and a patient monitoring system

BBC News: The bug bites NHS trust could not process appointments Canterbury Cathedral could not book tourist visits

The Sunday (London) Times: Bug bites Halifax building society informs customers of new policies valid "from 1999 to 1900" Oil company's crane would not operate - chip thought it was overdue a technical inspection

7. Source: The Australian

http://www.theaustralian.com.au/state/4227351.htm

(Excerpt)

Hospital systems fail test for Y2K By ANDREW McGARRY 6jan99

ALMOST a third of computer-related equipment in South Australian hospitals, including cardiac monitors and drug distribution systems, have failed the millennium bug test. The potentially disastrous results have forced the State Government to boost funding by $19 million to combat the effect of the bug, also known as Y2K, where computer systems fail to recognise the date January 1, 2000. Health Minister Dean Brown said the investment was needed because serious non-compliance problems had been discovered in several hospitals. Of the 22,761 systems tested in South Australia, 7042 were non-compliant.

8. Source: Fortune

http://pathfinder.com/fortune/1998/980427/imt.html

(If you are interested in the manufacturing industry, this article is for you. Below are excerpts from the article that were related to Y2K failures)

Industry Wakes Up to the Year 2000 Menace - April 27, 1998

Gene Bylinsky Reporter Associate: Alicia Hills Moore

Small wonder, then, that many plant managers and their bosses plan to stay close to their jobs over the three-day weekend when the millennium rolls in. Already they've had a foretaste of what could go wrong. A somewhat similar time problem--programmers' failure to account for the 1996 leap year--halted some production lines at the beginning of 1997, causing millions of dollars in damage. In simulations of the transition from 1999 to 2000, some factory robots, as well as computers that control electric power generation and transmission, stopped dead. Other warning signs have appeared. By erroneously interpreting a 00 datum as the year 1900, a mindless computer at a food company directed workers to throw out perfectly good products.

So for a long time manufacturing companies snoozed, including GM. When he arrived at the automotive giant a year and a half ago to take over the CIO job, recalls Ralph Szygenda, he was amazed "that most people assumed that the factory floor didn't have year 2000 problems. At each one of our factories there are catastrophic problems," says the blunt-talking executive. "Amazingly enough, machines on the factory floor are far more sensitive to incorrect dates than we ever anticipated. When we tested robotic devices for transition into the year 2000, for example, they just froze and stopped operating."

Leap-year snafus damaged production lines when programmers failed to account for the extra day in February 1996. At a small U.S. manufacturer of industrial solutions that prefers to remain unnamed, production ground to a halt on Jan. 1, 1997. Before workers could remedy the situation, the liquids hardened in the pipelines, which had to be replaced at a cost of $1 million. That caused late deliveries and the loss of three customers. Computers misinterpreting 00 year dates are already issuing demented instructions. Companies don't issue press releases about such bungles and can't be named. But one consultant tells of a U.S. freeze-dried food manufacturer that noticed its warehouse inventory mysteriously decreasing. Reason: The computer system, mistakenly reading the 00 expiration date on the products as 1900, had ordered stocks destroyed. Communications systems could crash too. The U.S. Army's Materiel Command, testing its PBX telephone systems for the 2000 transition, found that they ran fine for three days after the turn of the year, until accumulated date errors shut down the whole network.

9. Source Techweb republished from EETIMES

http://www.techweb.com/se/directlink.cgi?EET19981214S0057

(This article is about the embedded systems problem. I could not find any specific examples of Y2K failures. However, the last paragraph (below) does contain a general mention so I include it here)

December 14, 1998, Issue: 1039 Section: Embedded Systems -- Focus: Reliability And The Year 2000

Although much of the in-depth system testing is just beginning, there have been several reports of significant date errors found and corrected. Some of the test results would have been very serious if they had occurred during normal plant operation. It is hoped that this will serve as a wake-up call to those who operate their facilities and as a sobering reminder to the experts who believe the problems are trivial and the severity blown out of proportion to the actual danger.



-- Rob Michaels (sonofdust@net.com), March 05, 1999


Moderation questions? read the FAQ