Having read through many of the recent posts here and noting the growing concern over how premature Y2K success has been declared, I have been thinking back to my own experiences of Y2K testing over the past 2-3 years and how the disturbing feelings of something not being right that I had then are returning to me now.

I was involved, directly or indirectly with 4 very different testing projects which included Y2K as part of the testing phase. Although I wasn't directly involved in the actual Y2K testing on all of these projects, I had the opportunity to study the methods used and, importantly, the test data. In some of these projects, I arrived when the Y2K element was supposedly complete so I could also study the results of the testing.

Each of these projects involved collection of revenue in some form or another and covered a variety of platforms from legacy mainframes to hand-held computers. The very last one was more of a stores/personnel system but it covered an area of important government business.

My concerns over the testing focussed on the following particular areas and, if these 4 sets of systems are representative of systems on a global scale, then I think we have very real reasons to feel that the show is very far from over. Where it was possible and where I had direct involvement, I was able to address some of these concerns personally, within the time that was available!

1. Test Data - Without exception, all Y2K testing I studied used a very small subset of test data, and most importantly, did not apply COMBINATIONS of test data throughout the system under test. There was virtually no testing that checked on how Y2K dates would function simultaneously across different parts of the system; only single modules were checked and then, as I said, with limited data. No thought was given to the combined effects of non-Y2K dates and Y2K dates, i.e. possible consequences of date arithmetic.

2. Batch Operation - Some of the systems I worked with, involved automatic operations on data, each batch job being tested one at a time and again, with little thought being given to the effects occurring simultaneously across the system. Most frighteningly, these batch jobs, in production, involved substantial database update and even at the test level, only cursory checking was done to see that the data within these databases had been correctly changed. Furthermore, many of the problems in the testing revolved around non-compliant and compliant modules getting mixed up in the test environment. We are talking about hundreds and thousands of separate modules here, so how easy would it be to find a non-compliant module in production that had got there because change management had screwed up the promotion of the modules?

3. There was a heavy emphasis on convincing management that effective Y2K testing had been carried out by the use of wordy documentation that focussed on methodology but was very light on tangible test results (after all, management aren't really interested in that sort of thing are they?). Therefore, management perception of their Y2K effort failed to appreciate the shortcomings in terms of testing techniques and test data.

4. Some of the above concerns arose from the fact that those responsible for the actual testing were relatively inexperienced and/or lacking in knowledge of testing practices.

5. In the last piece of testing work I was involved in, perhaps the scariest of all, there was no actual testing, simply a module by module check, sometimes programatically, for all date references in the code in question. Again, no thought was given to simultaneous processing of non-Y2K and Y2K dates and, even worse, the actual code being 'tested' in this way was weighed down with specific processing designed to stop 2000 being stored as 1900 in the database behind the code because the operating software was not Y2K compliant!

As one final thought, on one of the testing projects, a database corruption, not due to Y2K but in the production environment, took SIX MONTHS before it was noticed! This was down to a batch job that had been screwing up the data, putting it in the wrong place in the wrong format. The mind boggles at how long it would take that particular financial institution to notice if, say, all or some of the dates were going in as 1900! Again, multiply this scenario on a world-wide scale...

I shall watch with great interest for symptoms of these concerns I have although, as has been already pointed out here on this forum, these sorts of screw-ups will have their lids very tightly sealed. Nevertheless, when company trading starts being affected, that will be more difficult to hide.

We certainly live in interesting times!

-- LongInTheTooth (yhl07@dial.pipex.com), January 02, 2000

Answers

Great post. Thanks. Fits in with my own experiences and those I have read about and corresponded about. Date arithmetic errors in banking calculations IMHO will lead to extremely serious imported data corruption in the banking system of systems.

I hope I'm wrong.

Watch banking over the next few weeks.

-- Andy (2000EOD@prodigy.net), January 02, 2000.

Thanks for your time in posting this...

We've heard that 10-20% of all systems have been worked on. Has that been the case where you've been Long?

-- PJC (paulchri@msn.com), January 02, 2000.

Thanks, yes, the number of modules actually tested out of all that could have been tested was certainly insufficient to prove conclusively that they all worked although I cannot give you a percentage. In my experience, it was a case of 'if that one works, and this one or two are very similar, in terms of processing, we don't really need to test them'. Once again, I take the systemics view that you cannot know what properties will emerge from a system merely by inspecting the individual parts. This is, in my opinion, one of the fundemental flaws in software testing practices today; far too much emphasis on disection and analysis, not enough synthesis.

-- LongInTheTooth (yhl07@dial.pipex.com), January 02, 2000.

Personally, I am treading tomorrow. I have been involved with the replacement of key order processing system with 7 independent mfg companies with one common customer. Spent last 2 month replacing system. Many, many problems. Now they are going to actually test for y2k issues on 1/3/99. As an aside I received two calls over weekend where home Pentiums PC went south. Non-bootable. I agree most of the big companies had people working over this weekend but I am sure very few of the small companies have. A good judge of the amount of failures on Monday will be to check out your local computer superstore. I was at one on Saturday, an the service guy had already received a number of "my PC is dead" calls. This thing is far from over.

-- Very Concerned (Veryconcerned@westol.com), January 02, 2000.

LongInTheTooth: Thanks for your thoughts on this. I share your concerns with many of these testing issues - particularly on larger systems.

In a recent special edition of Scientific American entitled "Extreme Engineering", an article appeared entitled "Gargantuan Software" that discussed the construction of Windows 2000, a truly mammoth piece of software. The statement that stuck out was that testing was far and away the largest part of the process - I believe the information presented was essentially "90-95% of the project was testing". I don't currently have the magazine here at the house so I can't give you a verbatim quote. Perhaps someone here has a copy of this article available and could post the precise quote.

As someone who has been bitten by focusing on component-level testing at the expense of systems-level testing, I have to concur that these are issues some organizations are certain to face during the coming weeks.

-- Arnie Rimmer (Arnie_Rimmer@usa.net), January 02, 2000.