"End to End" Testing

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Roger Altman started a thread:

Dick Mills on computer programming: If it ain't tested, it doesn't work

It doesn't surprise me that Dick Mills refused to give a personal tutorial on "testing" via e-mail. Others have tried explaining some of these concepts, Maria for example. But I think the question includes some common misconceptions, and points out one more reason why I'm fairly certain the error rate already experienced within systems is higher than that which will be generated over Y2k.

The question is in regard to "End-to-End" testing, or "Integration" testing. First, let's look at normal systems development. Assume System A, System B, and System C are being developed; they do not yet exist. System A passes data to System B, which then passes data to System C:

System A ===>data 1===> System B ===>data 2===> System C

The systems are developed seperately, and tested. data 1 and data 2 are defined interfaces. Whether actual files, or real-time communications, or "indirect" (reading the other System data directly), the interfaces are defined, but not yet tested in conjunction with the other systems.

Before being tested, virtually anything can go wrong. System A may not provide the information defined by System B; formats can be different; files may not be named correctly; communications may not be correctly defined; the list is virtually endless. These systems were put together on paper; they now must be put together actually running.

Before going to production, these systems must be tested in conjunction with each other.

Contrast that with the problem presented by Y2k. System A, B and C are already functioning in production, passing data. Now, assume System B is not compliant.

Two basic choices are available; fix System B to process rollover dates, or replace System B with a new system.

First, look at replacement. Every effort is made to make no changes to the formats and data defined for the interfaces data 1 and data 2. If no changes are made, then you are presented in essence with just a regression testing scenario; that is, can the new System B.1 process the data already being produced by System A, and can it duplicate the output to System C that is currently being produced by System B.

In reality, some changes are usually required. Most often it is in incoming data, but can be in the outgoing data as well. If format changes are required, it complicates the testing to a degree. This is the reason that "windowing" of interface file dates is so prevalent, to minimize the impact on existing systems.

If remediation is chosen, the process is even simpler. In reality, the only choice is whether or not to expand the date fields defined in interfaces data 1 and data 2, again highlighting the popularity of "windowing" dates.

The important point, though, is that no matter whether replacement or remediation is chosen, problems arising from the integration of the systems will occur when System B or B.1 is placed into production. If date expansion is used in either data 1 or data 2, and any of the systems are not setup to handle the new format, the interfaces will fail now, not on the rollover. The processing of the date fields, whether expanded or windowed, occurs when System B or B.1 is implemented. The format won't change just because the date is in the year 2000.

The only problem that can occur on rollover is whether or not System B or B.1 can process the date fields from System A, with Year 2000 dates, and whether or not System B or B.1 can produce Year 2000 dates for System C. This processing can be tested with testing of just System B or B.1 .

This is not to say Integration Testing should not be performed. But the importance is far different than during a normal "development" project. And in any case, the vast majority of errors that can occur due to notperforming "End-to-End" or "Integration" testing, have already occurred, or will occur when the systems are re-implemented. These errors are not waiting for the rollover to 2000.

-- Hoffmeister (hoff_meister@my-deja.com), October 06, 1999

Answers

In an ideal world, maybe. That's the whole point to functional encapsulation is it not? If a given component's functional interface is well defined and static, then, in theory, any given component can be modified or replaced without impacting the whole. This component- based approach to systems engineering has been the holy grail of programming for as long as I can remember.

Some companies even do a pretty good job of this today on some projects. During my time with Rockwell, working on the GPS project, this strategy was incorporated throughout our receivers, both in the software and in the hardware. It wasn't perfect and unanticipated results did frequently occur - often when we discovered that a given interface definition was incomplete. But it was a much better approach than what passes for systems engineering at most companies today.

I would argue that a great many systems are not well-engineered but are essentially 'enhanced rapid prototypes'. They get something that sort of works, then begin tacking on functionality until they reach the 'good enough' point. Even Rockwell was not immune from this tendencies. Among management at Rockwell there was a saying: "shoot the engineer and ship the reciever".

More significant, many complex systems today were evolved rather than engineered, mainly due to economic pressures. If you have two separate working components and wanted to merge then into a single system, often it was considered more economical to simply create a third component that 'glued' to existing two components together in some sort of ad-hoc fashion. Re-engineering the entire system was considered too expensive and unnecessary.

In all but the most critical of systems, it has been my experience that the level and degree of testing performed was more a function of marketing due dates and allocated budgets rather than of sound engineering principles. I have often seen systems shipped with inadequet testing, sometimes with catastrophic results (for the product itself).

To be sure, not all systems are 'rapid prototypes', but more than most people would think. Testing is expensive and often gets shortchanged if everything 'appears' to work on the surface. Interface definitions, if done at all, are often incomplete.

I would say that two of the companies that I have worked work, Rockwell Avionics and General Electric Medical Systems division get very high marks for systems engineering and testing methodologies. They looked at their engineering processes much deeper than many companies. But even then, unexpected problems arose.

Thus, I feel the concern over 'end-to-end' testing is certainly justified. What I think people need to keep in mind here is that not all errors are catastrophic, not all systems are mission critical, and not all bugs manifest themselves in the same way.

-- Arnie Rimmer (Arnie_Rimmer@usa.net), October 06, 1999.


You miss the point.

Whether or the components interfaces are "well-defined" or not, they are put into use when the component is reimplemented. End to end testing may identify indirect interfaces, and problems, granted. But those problems are not waiting to happen; they occur when the system is placed into production.

-- Hoffmeister (hoff_meister@my-deja.com), October 06, 1999.


That's a pretty good explanation, Hoff. Of course, it assumes that the remediation on all sides has been successful, does it not? Even more, it assumes that remediation has been attempted on all sides. Many entities, large and small, are doing nothing because screaming Y2K Scoffers have convinced them it's just a hoax.

What I think people need to keep in mind here is that not all errors are catastrophic, not all systems are mission critical, and not all bugs manifest themselves in the same way.

All of that is true, Arnie. And your observations about marketing and budget are right on. But I think people also need to keep in mind that some errors are catastrophic, and that some systems are mission critical.

But, I am just a Fear Monger for saying so. ;-)

Everything doesn't depend on computers. But some things do. Every computer failure isn't catastrophic. But some are. Every failure does not take weeks and months to fix, but some do. So, Collapse of Civilization? No. Bump in the Road? No. I added this part for those who are incapable of making these kind of distinctions. :-)

-- Lane Core Jr. (elcore@sgi.net), October 06, 1999.


uh, where is your evidence that the majority of systems have been fixed and put back in production? Some countries are just learning to spell Y2K. And most businesses in US are going to Fix On Failure.

-- a (a@a.a), October 06, 1999.

Yes Hoffy, the systems will fail now, not later. And guess what...the systems are failing now. You DO follow the news, don't you?

OWl

-- owl (w@a.com), October 06, 1999.



This would imply that the full functionality of a component interface is exercised as soon as the component goes into production. This cannot be what you meant. Errors in code often lay undetected for years until the right set of circumstances occur. The number of such errors are inversely proportional to the time spent on systems engineering and testing.

The engineering flaw in the space shuttle's boosters was not 'discovered' as soon as the shuttle was put into service. It took several fights and then the right set of circumstances (i.e. cold weather) to make this flaw devastatingly clear.

In an isolated system, such errors are 'discovered' and fixed all the time - this is standard systems maintenance. The reason that end-to- end testing (systems-level testing) is a valid concern is not because such bugs will not be repairable but rather the environment in which such problems must be addressed.

Fore xample, if I have a problem today with one of the systems I'm responsible for, I can get on the phone, speak to tech support, purchase an upgrade or replacement, and get it shipped to me in a reasonable amount of time. This is business as usual.

I simply don't share your same level of confidence that it will be 'business as usual'. A great many businesses today operate on very thin margins and tight cash flows requirements. There is a 'window of opportunity' for correcting serious problems, unique to each business, that will determine the consequences for that business of any significant disruption of it's own internal systems or those of its suppliers.

I'm not arguing for panic or hyteria but for increased fault tolerance. Systems-level testing simply represents a few bricks and some mortar in that wall of fault tolerance.

-- Arnie Rimmer (Arnie_Rimmer@usa.net), October 06, 1999.


Gawd, Hoffy STILL does not get it. He NEVER will get it.

MAYBE if fixing the Y2K problem were nothing more than expanding the year from 2-digits to 4-digits, you really could say that the problem was simple enough that remediated systems successfully in production NOW, successfully processing 4-digit years NOW, will reliably work in 2000 and beyond. (Well, until Dec 31, 9999 anyway.) A big maybe.

But, sorry, Hoffy; due to time and space, folks have had to resort to ALL KINDS of fancy-dancy ways to allow those two digits to also represent century information. These different methods may prove as incompatible as inches are to centimeters when you are on your way to Mars. Even Microsoft products that use windowing, for instance, have conflicting pivot points (the year selected to divide the 20th/21st century interpretation). Even if every piece of the puzzle is individually Y2K compliant, until you put them all together and do good ol' end-to-end testing, it is quite likely that things will fail in weird and wonderous ways once the clock rolls over to Jan 1, 2000. Which is why, once upon a time, it was fashionable to promote the idea that everyone needed to be done by Dec 31, 1998 so as to allow a-full-year-for-testing.

And yes, I remember Maria's "contributions" to this area quite well. I hope she will add her opinions to this thread, too. LOL!!!

-- King of Spain (madrid@aol.cum), October 06, 1999.

Gawd, Hoffy STILL does not get it. He NEVER will get it.

MAYBE if fixing the Y2K problem were nothing more than expanding the year from 2-digits to 4-digits, you really could say that the problem was simple enough that remediated systems successfully in production NOW, successfully processing 4-digit years NOW, will reliably work in 2000 and beyond. (Well, until Dec 31, 9999 anyway.) A big maybe.

But, sorry, Hoffy; due to time and space, folks have had to resort to ALL KINDS of fancy-dancy ways to allow those two digits to also represent century information. These different methods may prove as incompatible as inches are to centimeters when you are on your way to Mars. Even Microsoft products that use windowing, for instance, have conflicting pivot points (the year selected to divide the 20th/21st century interpretation). Even if every piece of the puzzle is individually Y2K compliant, until you put them all together and do good ol' end-to-end testing, it is quite likely that things will fail in weird and wonderous ways once the clock rolls over to Jan 1, 2000. Which is why, once upon a time, it was fashionable to promote the idea that everyone needed to be done by Dec 31, 1998 so as to allow a-full-year-for-testing.

And yes, I remember Maria's "contributions" to this area quite well. I hope she will add her opinions to this thread, too. LOL!!!

-- King of Spain (madrid@aol.cum), October 06, 1999.

(Sorry about the double post. Damn WEB-TV.)

-- King of Spain (madrid@aol.cum), October 06, 1999.

Arnie

Basically, I agree. But missing components is a function of the amount of testing performed, and is true of any system, and any potential problem, Y2k or not. That these errors lie dormant is testament to the frequency they are executed. And if this functionality is not tested during unit testing of the system, it most certainly won't appear in any defined test scripts for end-to-end testing.

Your Highness

Yes, I remember your, umm, interesting example from an older thread. That you had to make it up speaks volumes.

Not saying someone won't do something off the wall. They usually do. Just that the frequency of those occurrances is so small as to be inconsequential in the "big picture".

In almost every instance I can conjure up, different "pivot" dates for windowing present absolutely no problem, and won't for quite some time. The pivot date is based on the type of data. If Y2k truly presents such a danger, than arguing about whether or not problems might crop up 20 years from now is totally meaningless.

-- Hoffmeister (hoff_meister@my-deja.com), October 06, 1999.



Everybody stop worrying.

End-to-end testing will begin in 86 days...

-- Uncle Bob (UNCLB0B@Y2KOK.ORG), October 06, 1999.


Lane what a priceless response. Try to read the words again. Try to think about project implementation for Y2K. Try to understand expansion and what remediation entails. What is successful remediation? BTW how many lines of code have you remediated? How much work have you actually done in Y2K?

Hoff, nice explanation.

-- Maria (anon@ymous.com), October 06, 1999.


Yeah, Maria, it was about the best explanation anyone could offer. As if you had any idea what it meant.

Hoffy, NOBODY KNOWS how prevalent off-the-wall stuff is; there is a lot of old, old code running on old, old systems out there, which for the sake of this thread we are ASSUMING that have been happily remediated and unit tested, and that the only thing that they lack are end-to-end testing. If it were merely disagreement as to whether a windowing pivot point should be 28 or 50, and the only methods used were windowing and expansion, we might just be able to squeak by. But it ain't that way on ANY of these counts, and we are about to get squished.

-- King of Spain (madrid@aol.cum), October 06, 1999.

Hoffmeister,

I liked your original post and follow-ups. However, I doubt it will even begin to satisfy dear Dr. Altman.

After having had several debates with him on this issue, my perception of his stance on what constitutes an acceptable level of testing ("end-to-end" being his current euphamism for this) is that it can only be accomplished by using every component in a system in a manner exactly as it will be used in production after 1/1/2000. Furthermore, he has stated on several occasions that he does not accept this as being possible since it is impossible to exactly simulate the future operation of any non-trivial system.

Having read the thread you referenced in your original post, I do not believe that Dr. Altman has moved from a position he has held for over a year now: "I will not accept that anything is fixed and will work until you prove to me that everything is fixed. Furthermore, I will not accept proof that everything is fixed until it is tested to a level I have freely admitted is impossible to meet, namely that every component must be tested in exactly the way it will operate in the future, without exception. Therefore, I can reject out of hand any argument that things might not be disasterous."

Not that you haven't made a good argument, but I'm afraid it's made made to an audience that is clearly unwilling to hear any answer that might contradict pre-concieved notions.

-- Paul Neuhardt (neuhardt@ultranet.com), October 06, 1999.


Paul, that's my impression of Dr. Altman's viewpoint also. I tried to discuss testing with him but to no avail.

Your highness, your mental age ceased developing at the age of 12, didn't it? Yeah I understood every word and further wanted to point out that bridges would be another option for system B but didn't think it worth making the point. I decided to let it stand on its own.

-- Maria (anon@ymous.com), October 06, 1999.



Hoffy- how can the end to end testing be already done (as I see you claiming here) if some of the parties involved are not done yet??

For evidence that they are not done yet, check their statements wherein they claim they will be done before 12-31-99.

PLEASE let me know the answer to my question in a nice, simple reasonable format that poor, lil' ol me can understand.

Or not. Your call.

-- Brent James Bushardt (brentj@webt.com), October 06, 1999.


I bet Hoff talks to himself all the time because no one else will. Oh, except for some on this forum.

Less than 90 days now. What difference does this debate really make? Put the truth smack down in the middle, with evidence being valid on both sides, and a prudent person will still side on having preparations.

What is the point, Hoff?

:

-- sick of it (what's@the.point?), October 06, 1999.


(1) Some systems ARE failing now due to Y2K (and related) problems.

(2) Since not all systems are yet remediated (it's only October), the testing cycle will gravely shortened. Systems will be rushed into production (with assurances that all is well) long before they are ready. Yes, this happens with a lot of systems up for other remediation, as well!

(3) We will see an increasing level of system outages and failures during the next 12 weeks.

(4) Many date problems will surface in the early days of 1900..

-- Mad Monk (madmonk@hawaiian.net), October 07, 1999.


Since almost every polly I have ever run into on this forum has re-defined my position on end-to-end testing to the point of trivial nonsense, let me lay out my concerns as simply as possible.

What I'm asking for is evidence of ANY TYPE of end-to-end testing. Let's see what has been done to date. THEN we can debate how useful those tests have been. I submit that practically no end-to-end testing OF ANY KIND has taken place, no matter how loosely you define this term, BUT I'm certainly willing to review (and REVIEW in detail) any evidence to the contrary. If the pollys cannot provide this information then it is only reasonable to examine the likely outcome of simultaneously implementing systems designed to run as an integrated unit, BUT have never been tested in such a configuration. BTW, at least Hoffy dealt with this scenario directly, and his points were debated on this thread. Now let's tackle the end-to-end testing questing AS I DEFINED IT ABOVE. How's that? (How many doomers are ready to bet that the pollys will want to re-define my debating parameters because what I'm saying in not "relevant"?)

Roger

-- Roger Altman (RogAltman@AOL.com), October 07, 1999.


Roger

You are absolutely correct. The "likely outcome of simultaneously implementing systems designed to run as an integrated unit, BUT have never been tested in such a configuration" IS a large number of errors and problems, with or without "end to end" or integration testing. It is extremely relevant.

Where you go wrong is assuming this is some future event. It is not. It has been occurring over the last year. And has not, to the best of my knowledge, come anywhere near causing a system "collapse".

-- Hoffmeister (hoff_meister@my-deja.com), October 07, 1999.


Hoffy:

Except for the fact that you've got an incredible number who are sitting on the sidlines waiting to FOF, AND a very large number who are so far behind that they have not fully implemented their Y2k "remediated" systems anywhere to the extent that you assume.

I'm sure you're going to ask my to prove both statements, just as I can ask you to prove yours. So let me say again, each person must weigh the evidence that supports YOUR position NOT MINE. If ANYONE decides that you come up short factually or logically then he should prepare for trouble as best he/she can. To me this debate business is not only foolish but a terrible waste of VALUABLE time when the stakes are so high.

-- Roger Altman (RogAltman@AOL.com), October 07, 1999.


Roger brings up a very realistic fact: Time is very short. I'd rather base my Y2K preparation decisions on the assumption that the complex interrelated systems are going to have massive Y2K failures and problems -- and be pleasantly surprised to find that Hoffmeister was right all along. Rather than assume that things are going to work out just fine, then discover that it the wrong bet.

85 days.

-- Jack (jsprat@eld.~net), October 07, 1999.

Interesting, Roger.

So tell me, if it is such a waste of time discussing these effects, why did you request they be discussed in the first place?

-- Hoffmeister (hoff_meister@my-deja.com), October 07, 1999.


Hoffy:

So that the fence sitters can have a fair shot at making a decision to prepare or not.

I do the same thing in my neck of the woods. I've given well over a dozen Y2k talks, and our local cable station helped me produce a 15 segment, 7.5 hour Y2k video called "The Millennium Bug: A Threat Hidden in Plain Sight". It has been shown repeatedly in central New Jersey since late July. The station manager, who I'm proud to say is now a full fledged GI, continues to re-broadcast it on our local channel even now. At least one Y2k action group has formed after viewing the series.

-- Roger Altman (RogAltman@AOL.com), October 07, 1999.


Moderation questions? read the FAQ