EXEC SORT (or why end-to-end testing is important, or it's the little things that getcha)

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

We all, hopefully, know about the sorting issue with Y2K. Here's another view, from my office.

So we have this on-line transaction system, running on our non-compliant IBM 4341, that will not be converted to "new technology" by next year. We knew this, and have a century fix in place for this system. We expanded the history, and ran the system on a time machine. All looks good.

Next, we move the transactions to tha back-end system, the spanking new, Y2K compliant "new technology" stuff. Been doing 4 digit years here since year 1. You give us YYYY, and we'll deal with it. No problem.

We have this offload job, been running for years. Made sure the program included the expanded year. Created the file. Looks good.

Sent the file off to a SORT, a utility that comes with the OS. I can't get into details, but this is a financial application, where the file is sorted on State, Account #, DATE, time, and other stuff. The SORT CONTROL statement was not changed to include the expanded date. We were sorting on a 4 digit year, a 2 digit month, but NOT THE DAY! Very interseting results...

A simple fix. It wasn't even a "program change", just a "utility control statement". Tested the crap out of both ends. Burnt in the middle.

It's the little stuff...

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), September 03, 1999


I can visualize you looking at the reports or whatever: "What the f***!" LOL.

-- vbProg (vbProg@MicrosoftAndIntelSuck.com), September 03, 1999.

Thanks for the inside info, Sysman.

Unlike planes falling from the sky, power grid collapse, rioting in the streets, etc., your example of a y2k problem is boring and probably difficult to understand (but not for most of the folks on this forum, of course). It's so un-spectacular, no journalist would ever write a story about it.

And yet, it's a perfect example of the nagging, time-wasting, efficiency-draining effect of an unremediated y2k glitch. It won't by itself kill a company, but somewhere around, say, March 2000, the firm's top management will be scratching their collective heads and wondering why the hell they're operating at 70% efficiency and not showing a profit anymore.

If all those spectacular y2k problems do not occur (blackouts, riots, Klinton crowning himself Emperor-for-Life, etc.), the type of problems you describe, multiplied by thousands (millions?) across the economy, cannot help but cause a severe drop in productivity and efficiency.

If we avoid all the life-threatening crap (not a done-deal by a long shot), we're still staring down the barrel of a serious recession/depression and wicked-high unemployment next year.

-- rick blaine (y2kazoo@hotmail.com), September 03, 1999.

Exactly! And high unemployment translates to seriously reduced cash flow across the board which spells many other businesses that are otherwise OK - failing for lack of customers. I have been ranting about this gray area for months on several different forums and everyone seems to ignore it preferring instead to focus on the grid and utilities. From a personal perspective,there is a high probability that my DH will not be employed next year because he works for a manufacturing company based in Belgium and France - They are clueless over there. How many millions of other Americans will lose their jobs because they work for foreign companies at risk? This is the real threat to our economy - broken supply lines and ceased cash flow. So, even if the utilities work - there will be legends of families who won't be able to pay their utility bills. This is reason enough to prepare - grid or no grid.

-- April (Alwzapril@home.com), September 03, 1999.


This happens all the time with length changes. I remember vividly sitting in a small room with 5 other programmers for 36 straight hours doing a final test on a HUGE modification effort which involved changing the length of multiple fields. The final report of the test just WASN'T right, and we were ALL tired. The Director of IT was awaiting our results and we kept finding little things in various programs and restarting the job at Step 12...to no avail.

Finally, ignoring the applications involved, I looked at the job stream again. I couldn't believe we'd wasted two hours on that one. I made the change and burst out laughing....well...you know how punchy we get after working 36 hours straight. Once I'd retained composure I said, "Restart it at Step 10!" Yep...the sort in Step 10 had the problem, and the final test worked just fine.

The moral of this story is: Check the control language as carefully as you check the applications. Also, don't show up for work without soap, deoderant, a change of clothes, toothpaste and brush, and LOTS of vanilla air-freshener for those who didn't.

-- Anita (spoonera@msn.com), September 03, 1999.


Thanks for the excellent and very timely illustration of where the major Y2k problems are lurking. The major problems are, and have always been, in those old data processing systems not in the much more glamorous embedded systems. Yes there are problems with some embedded systems but they pale by comparison to the problems in the routine business applications.

Do you get the feeling that the high-profile infrastructure and embedded systems issues have become straw men which are taking the public's attention off the much larger volume of problems that remain in business applications? It has been my experience as a panelist that, if I answer embedded systems questions satisfactorily, the questioners move on to another panelist. The typical questions involve the standard infrastructure questions and then the standard embedded systems questions are asked of non-infrastucture providers. Then there is usually someone clever enough to ask if we have contingency plans. Much to my chagrine a simple yes is always an acceptable answer to that question even though all of us should be prepared to give specific examples of our contingency planning efforts.

I have yet to hear anyone at one these forums asked specific questions about testing, implementation, etc. I have yet to hear anyone asked to identify a "non-mission critical" system which won't be compliant and explain the potential impact to customers and employees. No one ever ask if the employees using the non-mission critical systems are also non-mission critical.

As you have illustrated Sysman the devil is in the details. Unfortunately, I believe most of the attention is still focused on the abstract. I just can't think of a good sound bite for sorts not working. My 5.5 severity rating held firm from about January of 1998(down from a 9 in January of 1997) until about three weeks ago. I am now at 6.5 thanks, in part, to the valuable information gleaned from this forum. Big Dog and I are moving in opposite directions while drinking our Y2k information from the same well, perhaps that is some evidence that the reporting is well balanced, broad and deep on this forum.

BTW, there are some new Y2k related PTF's for VSE environments as well the OS390 stuff reported. So far we have found none that would affect our implementation of VSE but it is still disturbing to see this still trickling out. Still, I don't expect we will see a headline in the Washington Post that reads: "Tardy PTF's concern system administrators!!" It's hard to imagine that attention grabber boosting sales.

-- Woe Is Me (wim@doom.gloom), September 03, 1999.

Woe Is Me, I wish you would post more often here. I think you always bring up a perspective worth thinking about, even if I may not usually agree with it.

-- Jack (jsprat@eld.net), September 03, 1999.

The end to end testing of one application with which I am acquainted was so rushed, in order to meet an arbitrary deadline, that:

It was not noticed that at least one group of remediated modules were not included in the tests.


A pair of widely used utilities that needed remediation, never even got on the list of programs to be remedeated.


Who knows what else was missed.

Meanwhile, since the remediation is "done", all or most of the remediation staff has gone on to other assignments.


-- Jerry B (skeptic76@erols.com), September 03, 1999.

Sysman --- Thanks for a superb post, alas. Hey, Woe! I suppose I am slightly more optimistic about certain things, but, hey, it's all relative, regrettably. I'm nuts enough that I still view Infomagic as a low, but real, possibility, and continue to expect a depression.

Honestly, as I think a alluded to in another thread last night (this might be worth a thread in itself, hmmmm), I am so, what, flabbergasted by the awesome spin-machine that I occasionally find myself blinking at how far out of the so-called "mainstream" my convictions are. "Can I really be right and they are all wrong, especially when the last thing I want to be is right?"

Also, as I have never denied, I am not in Y2K remediation myself (thank God), so I'm flying somewhat on vapors and past experience with the big boys.

Heck, I'm running a new-fangled Internet commerce start-up that does whizzy kewl things for its starry-eyed investor (and, yes, we are poised to eat lunch next year if the Net stays up. Sort? What's that?

If the Net doesn't stay up, the livestock, greenhouse, family dairy farm and midwife spouse should help us fend in the local economy. Barring Infomagic. There I go again.

Sysman's shop is the type that Hoff maintains is going to save our bacon because EVERYONE is doing it now. I only wish.

-- BigDog (BigDog@duffer.com), September 03, 1999.

Howdy folks,

We had another interesting SORT related problem with another system a while back. I think I posted it, but can't find it...

It was an old Cobol edit program. It added a SORT-KEY to it's output file. It was just called SORT-KEY PIC X(24). The documentation called it the same thing, with no reference to what the contents of the key was. We did the usual scan on the program, and it didn't seem to do any date processing.

But it did have a CALL to a routine called BLDKEY, or something. Sure enough, it returned a DATE right in the middle of the key. Well, the library had the OBJECT code for the routine, but no SOURCE! Thank god for those dusty old backup tapes. We found the source in an old ICCF library, a "private" library that every programmer has.

Bottom line, TWO DAYS to fix a problem that should have taken a few hours at most.

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), September 04, 1999.

Moderation questions? read the FAQ