In another thread, "Question for all Computer Professionals (FM, vidprof@aol.com, 1999-07-23)" the background of some computer professionals in relation to Y2K was discussed.

I disclosed some of my background with Y2K. To reiterate, the products that we write for IBM mainframe terminal emulation, gateway servers and related products have shown few bugs after months of testing. Also one of our mainframes is currently running in the Year 2000 without issue.

As I've said though, I DON'T WORK WITH APPLICATIONS. It looks like this is where many of the problems will crop up as well as possible errors with embedded logic.

My question to those professionals (and other professionals working on Y2K) is:

1. Do you have FIRST HAND knowledge of Y2K bugs. Can you describe them generally in a way that does not break confidentiality agreements. 2. Were the bugs you fixed significant, in that they would have caused a system failure or serious data corruption/loss? 3. If (2) is true, would the bug have caused a cascade or 'domino' kind of failure, or would there some kind of work around? 4. Statistics on the above.

Thank you, Bryce

-- Bryce (bryce@seanet.com), July 24, 1999

Answers

Last week I was involved with an interesting Y2K test for a large hospital in a North American city to test their patient information system. What made this test noteworthy was that this particular legacy system had been fully remediated and tested by the vendor and certified as Y2K compliant. Most of my own Y2K consulting work has been on the vendor side, so this was a good opportunity to see what types of things would slip past the programmers and QA team, but would be picked up by real users in a production type of test environment.

Results:

The January rollover occurred normally, but a number of bugs were logged in "April 2000" of the test. One of the patient locator screens which allows an admitting clerk to create a hospital admission using the data from an outpatient clinic or day surgery visit showed the patient visits in the wrong order, i.e. they are supposed to be displayed in descending order within clinic type, but some of the most recent visits were coming out after the oldest visits. Not a serious problem, but pretty annoying for the clerk who has to scroll all the way down to the bottom to pick up the correct visit.

This problem was tracked to the way that pointers to visits were stored in the database - the visit serial number encodes the two digit fiscal year, and as this institution starts their fiscal year 2000 on April 1, 2000, visits after this time were picking up a "00" and so the pointers were getting stored in a different order than the display routine expected. I can easily see why this was missed by the programmers in the code check, and I think it got by QA because they did not build a sufficient volume of visits after April. Also not that easy to spot, as the primary sort is by clinic type rather than date, so the dates jump back and forth anyway. But a user looking for a specific visit would note it right away.

The next problem identified was in a module that codes statistical data for a series type of program, chemotherapy, for example. At one point the user got the error "data for period prior to 87-01 has been purged". What was happening is that the date edit routine was comparing a treatment date's fiscal period, in this case 00-01, with the last date that data had been purged from the system. The programmers had indeed caught a Y2K error in this part of the code and changed the comparison to a 4 digit year compare. But in this particular test environment, a purge had NEVER been done, so the last purge date was null. In this case the code fell through to setting the purge date to an arbitrary default period, i.e. 87-01. What was missed was changing this default to 1987-01. As it turns out, the hospital does do regular purges of data in their live system, so it never would have happened in "real life", but it was still nice to clean it up. In further testing three more occurrences of the same bug were found in related applications. So in total, five bugs were reported.

None of these issues would cause disruption to normal hospital operations or patient care, but they are the kind of things that would rile the users. The second one in particular, I can just imagine the poor person in technical support getting an earful - "This stupid system, it wasn't even INSTALLED in 1987"! Based on the amount of time it took my team to resolve these issues, I would estimate that technical support in a production scenario would be able to correct these problems in 3-5 hours in each instance.

I hope this level of detail did not bore you too much, but I think it is a pretty good indicator of the type of stuff that will sneak by the testers.

-- Computer Pro (first_minister@hotmail.com), July 25, 1999.

Computer Pro, thank you *very* much for posting some real life examples! As caregivers working in hospitals/clinics/dr offices and being in a different one every day the last two weeks, we can assure you any glitch will ripple into hair-pulling annoyance ;^) The system right now isn't sailing along perfectly, and the hours upon hours waiting for info, lost records, uncoordinated drug orders, etc translates into expensive time wasted.

Gotta rush our pt to the VA Geri Psyche Emergency Rm now, CYA.

-- 2 worn-out Cascadians (he'sdriving@us.nutstoo), July 25, 1999.

I'm a business analyst/tester in the software industry, related to property-casualty and life/health insurance. I was an insurance broker and underwriter for 12 years before getting into the software industry. I am not a programmer, but much of my job involves the analysis of how software interacts in the business world, and what a client's requirements are.

As a GENERAL rule, insurance companies, especially life companies, have the Y2K bug at least in a cage. The inherent nature of policies looking ahead into the future requires compliance. Many property casualty companies, however, and agents, rely on state agencies to supply downloaded data from MVR, CLUE, and the MIB. If the state computers are not operational policies could still be issued but there would be no way to determine eligibility. On the commercial property-casualty side, this could be serious, as certificates of insurance are required on construction sites before work is started.

I worked with one company in NYC that writes commercial and small rental polices. The board of directors had NO idea what they were doing. The program to upgrade their systems is about 18 months behind schedule. They may be compliant; I'm no longer at the company that provided services. However, once they decided to spend the money to upgrade from mainframes, they wanted the luxury of planning a new efficient system. Figuring out what they wanted has cost them serious delays. They were using mainframes when I was there.

The big problem now is on the agency and medical care provider side. Our company estimated that 25% of all agencies will go under as a result of Y2K. Many agencies are run by old timers, running DOS programs on 286s. Our support staff would make outbound calls to tell them its too late-sell the agency, and also send out certified letters telling them about their non-compliance. Many would hang up the phone, thinking we were trying to sell them software, or think that Y2K is marketing hype. This has serious implications-if a construction company needs an insurance certificate, work cannot begin until that document is secured from the agent. Should the agent not be able to access files, that firm is out of work until the paperwork can be generated. Most of the managers at my company are techies, and in general (no offense intended) do not have a realistic view about how the free market/business works. Most say, So, they cant find files for a few months. Big deal. It is a big deal if one is a carpenter out of work, or in a car accident and the agent cannot process the loss report. Yes, the large agencies will buy out the non compliant ones cheap, but this takes time and the information has to be transferred to the new databases. My former employer, a large agency, was on top of this issue from the beginning. When they moved to a new, large computer system, it took months to get all of the bugs out. Most products are sent being beta-tested on the companies and agents-it would take too long to do extensive testing. To my knowledge we have not seen "cascade failures." A friend of mine in QA and I tried running some post-2000 numbers into the system the old agents use. The system just hanged and required a restart when the effective date was used, and just blanked out the output from the printer when the date was used for a claim record.

One major US health insurance firm has serious problems with Y2K. They were behind schedule and sent work to a well-known computer/software manufacturer. Unfortunately the other firm used all newly trained programmers and none of the code was documented. When the work came back and modifications were needed, all of the outsourced code had to be scrapped. They are having problems with the claim proccessing software. My best friend is insured with this company, and is recovering from cancer. Currently the company is over a year behind in making 35K worth of payments and he has hired an attorney to handle the collection agencies and go after the insurance company. Multiply his problem by about a million after Y2K. My experience dealing with some medical practitioners, and the non-compliance among medicare providers including hospitals, leads me to believe that Y2K will be a disaster for anyone with chronic conditions, taking medications, or on life support. Even is the infrastructure for care is there, the loss of medicare will lead to catastrophic bankruptcies among the medical profession. How long can a medical institution function with a 50% cutback in revenue? How can, by default, a pharmaceutical company stay in business without that income down the chain?

The most disasterous aspect of Y2K from the companys standpoint is that of liability. Agent and Broker magazine ran a 4-part series on this issue. I include this issue because it is a bug, like Y2K; only this one is a legal bug. There has been talk about banning lawsuits, and have applauded that. But this action would also have very negative consequences. Say someone was driving drunk and hits your car, disabling you; now you can't collect damages. The same analogy works in the business world. Stay with me here.

There is some disagreement as to if Y2K is fortuitous. For a claim to be covered, the event must be unexpected, or fortuitous. But, Y2K is NOT fortuitous, it is known. On the other hand, there is no Y2K exclusion in most polices. Is is covered? Many companies are still grappling with that issue.

Lets say an insurance company has solved that issue, Company A. Company A will insure a client if he can show proof of compliance and compliance from vendors. An agent, covered by malpractice (or Errors and Omissions) insurance by company B, sells a policy to Acme manufacturing firm that shows compliance from all vendors. The agent tells them in good faith they have Y2K coverage because of these statements. Come 2000 a consumer suffers a loss from Acmes product due to non-compliance. Company A finds that the real cause of the loss was Acmes widget supplier, Amalgamated Widgets, and declines to cover the loss for whatever reason. (Acme gave faulty information, Amalgamated was a new supplier undocumented on the policy, agent didnt disclose, whatever). Now Acme sues Amalgamated and the agents E&O policy, company B. The E&O company now sues Amalgamated Widgets for misrepresentation...you get the idea. The example the magazine gave was similiar to this one and was pretty complex.

Companies will fight for years over who settles a $600 claim. We had a case where a client had a small hairdressing business in the house. We strongly encouraged them to spend a few hundred extra dollars and put the business and home with the same company, and put it in writing. They wouldnt to save money and placed each risk with a different insurer. A friend came over for a hair cut and fell, $600 claim. Two years have gone by and the companies have not settled this, one says it was a personal visit the other says it was a business call. The client was, when I left, now threatening to sue the Agents E&O carrier.

In the world of health insurance, companies will fight forever about minor payments if there was a primary and secondary insurance, or failure to get authorization. Once again, this will apply when phone lines are down, a company goes insolvent, etc. The entire health insurance. pharmaceutical, and provider system is a time bomb with 5 months to go, IMHO.

-- Retroman (retro50@agapeis.net), July 25, 1999.

Obviously: Don't get sick, don't get hurt, and avoid automobile accidents.

-- Tom Carey (tomcarey@mindspring.com), July 25, 1999.

Part of good preps is to take care and stay healthy now. Plus, anything you know that might need fixing, do it now.

We're on a surgery recovery case at the mo, not hospice (although it might become that), and we've been dealing with multiple Drs, clinics, hospitals for same patient; the ball has been dropped many times.
This with the infrastructure up and purring, summertime. If this were January, our patient would have passed on 4 weeks ago. There are so many elderly patients with multi-system failures. Takes a lot of work to prop each malfunction up and prevent disastrous cascading drug interactions, etc.

Watching diseases ravage bodies makes us a little more aware in the overall GI arena ;^)

3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0 3~0

-- Ashton & Leska in Cascadia (allaha@earthlink.net), July 25, 1999.

Thanks ComputerPro and Retroman,

I don't go for the doomer vs. polly conflicts.. The information you provided really helps a lot of us to get a handle on this. A bulletin board is great for this.

Any more professional stories out there?

-- Bryce (bryce@seanet.com), July 25, 1999.