Hamasaki: Enterprise systems: the real Y2K issue

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

> > cory hamasaki wrote in message > >OK, it's Polly Quiz Time. > > > >Why is this one being cross posted to c.s.y2k? > > > >On Wed, 27 Oct 1999 22:19:09, mandl_wisc@my-deja.com wrote: > > > >> We are upgrading MVS 5.2.2 with os390 2.7. After examining the PSP > >> bucket, we were not able to locate any information referring to > >> connecting these two OS's together. We are concerned about connnecting > >> the catalogs. Has anyone been at this particular level and performed > >> this massive jump in releases? If so, were there any toleration PTF's > >> or other special maintenance that had to be applied?

Yo bob,

The point of the posting was this. I've seen a number of posts from pollies who believe they understand Enterprise Scale Computing. In most cases, they appear to be management types or camp followers who have acquired some of the jargon and have fooled themselves into thinking that because they've heard some of the terms, that they understand the issues.

In addition, I've seen comments from them that clearly indicate that they *don't* understand the issues. In those cases, they remark that "I'm snowing them" or "am still in the old days" and that "they've moved on to newer technologies or greater understanding."

These pollies tend not to infest c.s.y2k, although a couple have swarmed out from TB2K and de-bunk-house.

I don't want to be cruel to them. I have a number of friends who fall into this category and, for the most part, I avoid technical discussions with them. Two of them have said that "they could do technical work, if they really wanted to but they don't."

The fact is, most of the posters in c.s.y2k do not understand large complex systems. This problem is grossly underestimated.

When these fail, it will not be possible to repair them. The heart of the matter is the level of effort. Citigroup spent almost a billion dollars. SSA has been working on Y2K for 10 years. This is the right order of magnitude for addressing Enterprise Systems problems.

Note: This is *not* the same as the E in ERP. ERP is an affectation. It doesn't drive 3890's, perform acturarial calculations, or print food stamps. ERP/SAP is a side issue.

The non-generic, mission critical, custom systems were built by hand, using tools like OS/360 MVT, IMS/DB, VS-COBOL, Assembler-H, PLIX, NOMAD, BDAM, Telecommunications Access Method, ACP, VM/370, etc. This building took place between 1965 and 1985.

There were similar spurts of large scale software engineering in the PDP/VAX arena, the S/3, S/3x, the Series-One, Data General, Wang, Honeywell, GE, CDC-Kronos, Univac, worlds.

Everything since 1985 is an affectation, a layer of technology a few molecules thick, a fascade on the underpinnings of a computing infrastructure that our pollies are only slightly aware of.

When this infrastructure fails, it takes the facade with it.

Did all 50,000 shops running IBM-style mainframes expend Citigroup appropriate funding, start work when SSA did? Or did a few, say, 49,500 start at the last minute in Enterprise Systems Time, in 1998 or 1999?

How many are so far behind that they will not be able to repair their systems before the organization itself collapses?

How many have yet to start? How many are running software that is far, far backlevel? Do the pollies have the technical awareness to realize that there are backlevel mainframe shops?

No one knows what will happen. With only 64 days left, there is still not good information.

You're personally in pretty good shape, perhaps not as good as milne, tim, or the Baron but better than 99.9%. Most of the U.S. isn't ready for even a 2 week infrastructure failure.

This mess is a great mystery. How many of the 50,000 shops are like Citigroup and SSA. I expect both to experience internal Y2K failures. A safe statement since SSA already has had them. I'm hopeful that both will be able to repair their Y2K problems in short order. We've already seen problems in the best-of-the-best. What about the average, the worse, the worse-of-the-worse?

I wonder about the others. For me this is an unknown. Paradoxically, there are commentors on Y2K who have absolutely no understanding of the technology, risks, or issues but who cling to an odd meme that this is about round numbers.

This is their delusion. You've seen them, they discuss planes falling from the sky, microwave ovens, and VCRs. While anything with a computer that processes dates is at risk, the threat is with the large older systems. They don't talk much about these systems because they don't understand.

cory hamasaki http://www.kiyoinc.com/current.html 64 Days, 1543 Hours.

-- (pollies@pcweenies.com), October 29, 1999


I think Cory is wrong about Embeddeds. That is where the catastrophies will be.

But what if we are BOTH right??? These computers are Digital Self Destruct (DSD) devices that will take the entire infrastructure down!

-- K. Stevens (kstevens@ It's ALL going away in January.com), October 29, 1999.

I think the "pollies" reactions to talking about large computers is caused by the very real and massive gulf between "Rolling your own" computer design, before the age of books on how to build one; and buying one off the shelf "Ready to go". You can not understand what you have not seen, and no amount of talking will produce the "AH HA! Now I understand " effect. A guy writes the code. Another guy walks up and says, "Neat stuff. Just what I need. Can I have a copy?" The code cutter says, "Sure! That why I made it. I could no longer stand watching you hand punch Hollerith code onto cards." Well, Thanks. I buy java next time, OK? Donuts? Sure! O.K. "Oh by the way, what do you call this thing? "I call it Basic Tape Access Method,...BTAM for short." "BTAM is is. I'll spread the word." Guy walks away. Code cutter continues to cut code with the mind running in SIMO mode as he thinks about donuts. Loyalty!. Kindness! Gentleness! The thrill of creation! Play this story 30,000 ways on 30,000 days and you end up with all the buzzwords in the legacy systems.

The polly side is like this. Beat your mouse a thousand times, I.P.O. your webmaster lines, get your coin in foreign dime, a C.I.O. ...Just in time, retire somewhere, it's no crime.

Set Channel address word CAW device I.D, ccw word chain set Channel command word one CCW1 Seek bb,cc,hh set " " " two CCW2 Search ID equal. set " " " three TIC Transfer in channel set " " " four Read CKD count key data.

9c000500 SDV fa000bad HLT

The polly problem is simple... On a clear disk, you can seek forever. bye

-- Art Soukup (asoukup@mscomputers.net), October 29, 1999.


-- Old Git (anon@spamproblems.com), October 29, 1999.

Wow, I love it when you all talk dirty. Seriously, it is scarey to think that the pre 1985 BTAM, VTAM, SNA legacy systems that databases were developed for and then modified by a whole lot of folk who didn't document well are still the backbone of the enterprise. John Q public does not get it that a lot of this was developed by three guys in a hot tub smoking a joint who neglected to document the what and how. From what I read, it is real easy to miss a date and remediation just introduces more complications and will not be 100% correct. Chronic Y2K malaise is the least of the concerns.

-- Nancy (wellsnl@hotmail.com), October 29, 1999.

Actually it wasn't 3 guys in a hot tub with a joint, it was 5 or 6 guys sweating a LOT, living on Coke, Tab and Fresca, with chips and chili and GALLONS of the worst coffee seen outside an ER, who put in 20 and 30 hour days, and when they COULD see straight they couldn't remember WHY that particular variable had to be spelled that way but by GOD it WORKS that way so we ain't gonna change it.

Chuck who HAS seen the formation and the aftermath and, like sausage, if you love it don't look closely

-- Chuck, a night driver (rienzoo@en.com), October 29, 1999.

kstevens: I think Cory is wrong about Embeddeds. That is where the catastrophies will be.

Yes. I could be wrong. I might be a polly on embeddeds because that is not my area. I have done some biomedical controller work and spent a year or two on Z80 assembly language 20 years ago.

I'm not a channel IO person like Art. The clue is how he dashed off CCW, CAW, TIC. These stand for Channnel Command Word, Channel Address Word, and Transfer In Channel.

I've debugged Channel Programming problems using line traces, GTF, and standalone dumps. This was in TCAM and in HASP RTAM.

My specialty is large Enterprise systems (and donuts). People like Kosky, the TeeVee news folk, and the assorted handwavers might have a good understanding of donuts but they have no idea what's going on with Enterprise systems and the mainframe world.

-- cory (kiyoinc@ibm.XOUT.net), October 29, 1999.

I think the real terror in these legacy systems are not just database collapses, etc., but the real degree of modifications that have occurred and a subject that no one has approached on this forum that I can remember seeing:

The majority of large shops, IBM or otherwise,have had such an influx of personnel pass through their doors, that no one, and I repeat no one, really has a clue what some one did to one module or another until it fails. And documentation, BWAHAHAHAHAHHAHAHAHAH, that's a joke in and of itself! So all you pollies who think that these programs written for remdiation have worked, all you morons who think that Y2K is a joke, remember something I saw in an earlier post:

It's the code stupid.

Gold is looking better by the second, John

-- John Galt (jgaltfla@hotmail.com), October 29, 1999.

Best line of the year, thanks Art!

"The polly problem is simple... On a clear disk, you can seek forever."

It will be interesting, in retrospect, to see how much reported Y2K remediation ever touched these systems at all as compared to how much "fixed" the "facade". That's a great analogy, Cory, though I like to think of it this way ---

Remember Mike Mulligan? He and his trusty sidekick dug a hole and found themselves in the middle of it. What to do? The town fathers turned him into a utility man and the steam shovel into a furnace. Over them they built a great building. (Computing: 1945-1985).

Now, they furlough old Mike and his gang off (too expensive).

Then, they build all sorts of cool, nifty, Rube Goldberg-contraptions up and around the central building, some connected by courtyards, some by walkways, some completely disconnected. Various of these structures have their own "utilities" and can more or less take care of themselves. But when you trace back their ability to function far enough, you find that underground boiler room, still manned by Mike- alikes, but largely forgotten by the surrounding world. (Computing: 1985-1999).

In fact, the surrounding world thinks they can get along just fine should the core heating plant .... blow.

We'll soon find out.

-- BigDog (BigDog@duffer.com), October 29, 1999.

This was posted by "Slammer" on the STOP THE PRESSES ... thread. It also belongs here, IMO


Flint, I respect your opinion. I'm not a super doomer but a middle of the road guy. I do not, however, believe we have the levity for unfounded optimism at this point. I do not work at the microcode level, however I operate in a multi-threaded client server data base acquisition and processing environment. On a previous thread I put forward a code fragment with a preported Y2K problem in a challenge to one guy who was a pure pollyanna. I do not believe you are a pure pollyanna but have a reasonable but slightly different point of view than my own. It was multi-threaded code that had timer threads monitoring the health of a data acqusition process. It worked fine in test, but failed when we went to production because the hardware operated at a faster speed. This was because the powers that be in our organization decided to implement different hardware than that used in the test lab, causing a real time problem in the thread synchronicity. This was because the testing wasn't deep enough, and the coordination between Distributed systems, and development/testing wasn't cohesive enough. The result was I've seen a management information system providing realtime information on 180,000 employees fail. Thanks to networking a solution was available and implemented in a week. Also the system wasn't critical to the business operation, and so wasn't readily apparent to the customers. My point is that in best case situations we never have enough time and testing. In the Y2K situation we are being forced by time constraints to roll out upgrades with even less than insufficient testing because comittee's are changing the infrastucture platform even while we are rolling out. Hence the increasing failure rate we are seeing. I believe we will make it thru this but this is a deep subject and pure pollyanna's don't understand complexities of this nature.. Also there appears to be a disconnect in the minds of management between hardware and software. A code regression test that produces zero differences between expected and actual results in test could be influenced by changes in hardware/OS at rollout time. Nothing this large (whole U.S. rollout over approximately same time frame) has been attempted in history. If anything we need more broader testing and deeper, unfortunately fate and time has denied us this luxury. A problem such as I've mentioned would never have been picked up by an IV&V code factory walk thru, but it happened. Good procedures are a must here. The U.S. has spent the money (200b) and the fix is in. The problem has been addressed, but it is inevitable that were going to have problems. The IV&V guys should be commended. The ones I've worked with have humbly tried to excersize diligence in the face of a bad situation and have done thier best. every day more remediated systems roll out and are being impemented, and I've really seen people pull together in a bad situation. Lets hope and pray that it is enough. I say to all the pollyanna's out there that I understand why you would say what you say but don't you really think the people have a right to know. You super polly's are making the doomers come down on moderates who might present a reasonable view to support your argument. (By the way, if you don't know what a regresion test is you probably shouldn't be stating you case as to wether you think the problem is fixed or not)There's no logic in these arguments if they are heated. In any event the super polly's (some of whom are most likely plants to disinform and break up public opinion) seem to have muddled the view point enough to prevent preparation by 90%+ of the population so its really to late now. I would just like to point out to the pollyanna's that if we get through this with only a small number of catastrophies that it was because of the dilligent hard working IT guys working at all levels.. Microcode hardware integration, database processing, networking, business cycle programming, who have stated facts that you've had to shoot down over the years, that you made it. Slammer...

-- BigDog (BigDog@duffer.com), October 29, 1999.

Yo, Cory,

Not my intent to pick a fight over Embeddeds versus Enterprise systems. Just that if we are both right, this thing will hit with much more of an impact than my 5+ but not a 7 current forecast.

At any rate, the controversy should ALL go away in January as we survey the wreckage.

-- K. Stevens (kstevens@ It.s ALL going away in January.com), October 29, 1999.

Every network interface/router/bridge/mux is an embedded controller application. All channel equiopment is embedded. There's stuff out there under the desks and in the back roms that NO ONE has a clue about and is always FOF by the company who sold it (if they are around when the F hits the FO time.

Screamin' daemons!

-- ..- (dit@dot.dash), October 29, 1999.

I feel REALLY SORRY for whoever is trying to remediate some of the code I wrote...especially that COBOL system that called Fortran subroutines, which, in turn, called COBOL subroutines (I didn't design it, though...just worked on it)...

And I really wonder if any of my really OLD stuff is running somewhere, maybe "coverted" to COBOL...the AUTOCODER code.

Yeah, DOS, MFT, MVT, MVS, DBOMP, CICS, VM/CMS, DOS/VSE, and about 26 (mostly assembly or machine) languages...I remember them all too well.

-- Mad Monk (madmonk@hawaiian.net), October 30, 1999.

My comments may not belong here.

I'm a graphic designer with only enough of an understanding of the internal workings of a personal computer and an OS to be either dangerous or godlike. I identify well with the whole idea of putting something together and then being astounded when "it" works.

Creativity is my life.

I have also known the absolute panic when my system just goes dark while I'm working on a deadline. My little, work-at-home enterprise must rely on that system in order to survive and this is only at the desktop level! I've lived fix on failure and only by the Grace of God have I managed to survive.

I understand fully the whole idea of "the facade."

In any graphics program you build layer upon layer. When you paint on a canvas, you do so layer, upon layer. When you build a great empire you build layer upon layer. The person who looks upon the finished work has no real idea about how a digital art file is created or why the strokes of a brush went in a particular direction or just how the foundation under a great structure is conceived and built. They only see the illusion of "the finished product." Most don't understand that the final piece was never viewed as really "finished" by it's creator.

Creativity is like driving a thousand miles an hour into a brick wall just to see what will happen. It's a serious of accidents and experiments and great energy which all seem to converge and join in some strange, magnificent rush and harmonic balance.

There are always reality checks.

Sometimes, a simple system crash because of some totally unrelated task running in the background can destroy days or weeks of worth of work in less than a heartbeat.

Amazing, priceless works of art have been discovered under layers of oil paint on a "recycled" canvas. I suppose someone just didn't appreciate the value of the original work at the time of it's creation.

And, it's no mystery that, on this planet, whole empires have been built upon the ruins of fallen empires.

Everything since 1985 is an affectation, a layer of technology a few molecules thick, a fascade on the underpinnings of a computing infrastructure that our pollies are only slightly aware of.

Thank you Cory, and Art, for such excellent poetry.



-- Michael Taylor (mtdesign3@aol.com), November 02, 1999.

Mainframe Programers Sanctuary Rogue Valley Oregon

United We Stand Divided We Become Slaves

Bookmark-y2k-Location By Default (b'rit)

-- (tim@cdsnet.net), November 04, 1999.

Moderation questions? read the FAQ