FAILED INTERCONNECTIONS - A First-Hand Experience

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Took the day off to help my wife with T-day preps. While the cranberry sauce is simmering, thought I'd pass this along. This is my first post here, so please forgive if the links below don't work - just tried plain HTML tags

I was recently involved in attempting to fix the failed interconnection of two systems which were previously declared "Y2K Compliant" by our local Public Utility District. I work for the County government as an RPG programmer. On Nov 3 we received an urgent call from PUD asking for help (they have very limited staff). They had attempted an interconnected Y2K test, and it failed. The two systems involved are for billing, NOT power generation/distribution. A JD Edwards system is compliant 8-digit. A custom system (also compliant) feeds 6-digit dates (and other data) into JDE. There are several points of interconnection. PUD has since hired a consultant and they're working on it (< 24 workdays left!).

My point is this: If a small rural PUD was blindsided by interconnection problems on Nov 3, how many other organizations with *compliant* systems will be blindsided on Jan 3?

US Bank and the State School Commission just got around to testing their interconnections to the courthouse *last week*. How many others out there are pushing the interconnection testing this close to the deadline? Or not testing interconnections at all? Or don't know how many interconnections they even *have* (IRS, SSA)? Can anyone spell "Beech/Oleson Pain Index"?

One last question for the forum: I've seen DB2 interconnection problems firsthand. What about embedded system interconnections? Are they more/less likely? I've read most of the material from IEEE and it's quite striking. But IEEE's Open letter to Congress seems to focus more on mainframe systems than embeddeds. Anyone out there checking embedded interconnections for crying out loud?

Just a country programmer with 65 chickens and a generator

-- RPGman (tripix@olypen.com), November 24, 1999

Answers

Gee, maybe THIS is an example of why everyone was supposed to be "finished with Y2K remediation by December 31, 1998, leaving a full year for testing".

Gawd.

-- King of Spain (madrid@aol.cum), November 24, 1999.

Beat me to it KoS!!

Ray

-- Ray (ray@totacc.com), November 24, 1999.


To quote Raiders of the Lost Ark: "We've got people working on it, Top people."

My esteemed employer had a "expert" programmer who said ic a y2k crisis meeting we'll just test two systems on two separate O/S's off- line independent and then just figger that they will work together with all the other interconnections during the rollover.

World class remediation, Assume and Hope.

-- squid (Itsdark@down.here), November 24, 1999.


That's okay - it's only billing.

No problems, mate! Just pothole before the-bump-in-the-road towards (no) paychecks (and no receipts!) in Jan and Feb......

You know - I'm beginning to wonder if the "run on the banks" that is so prominently feared (for cash) isn't going to be overshadowed (in the real world) soon after by massive (millions ?) of accounts getting overcharged/undercharged/not charged/no automatically deposited/incorrectly automatically deposited.....by all those millions of users (like the students in FL, teachers in Detoit, Philly, DC, .....)

-- Robert A. Cook, PE (Marietta, GA) (cook.r@csaatl.com), November 24, 1999.


I didn't know Rocket Propelled Grenades were programmable.

-- nothere nothere (notherethere@hotmail.com), November 24, 1999.


(R)eport (P)rogram (G)enerator -- an archaic language about as old as COBOL, used on mainframes and (in my case) AS/400 systems. Our 911 system, Treasures, Assessors, and Auditors systems are all written in RPG. It's a blast from the past!

Just a country programmer with 65 chickens and a generator

-- RPGman (tripix@olypen.com), November 24, 1999.


WOW! Got my question re embedded interconnections answered in the post below on the NIST. First IEEE, now the National Institute of Standards and Technology! Interconnections is turning out to be a REAL Achillies Heel. All the trees may be "fixed", but for forest is going up in flames!

Just a country programmer with 65 chickens and a generator

-- RPGman (tripix@olypen.com), November 24, 1999.


Sorry, I'm not a programmer at all but a project manager for ERP systems knowing about your problems. PROGRAMMERS: What's about interfaces between more than three different 6digit systems while all are compliant by "Windowing", but all with different "Windowing" setups (like one have the break 20/21 the other 50/51)

-- Rainbow (Rainbow@123easy.net), November 24, 1999.

RPGman,

I've always believed that the biggest problems during Y2K would be caused by EDI and cross communication of embedded systems. I've worked with EDI quite a bit both within my systems and intersystem with State and other local agencies.

I'm an officer with the local Sheriff's Dept. that has been doing the Systems Admin/Programming for the Dept's multiple AS/400s for the past 7 years. I got after our vendors early on (1996) re:Y2K and we've been remediated and tested on all but 1 application since early this year. The one last application is the 'business' system foreclosures, garnishments, etc. and is being upgraded this weekend (but I doubt it will be ready in time). Luckily, that application doesn't pass any data, it only receives, so if it's not compliant it won't bomb out the rest of the applications.

All of the inter-program interfaces were programmed by me using RPGIV (a very MODERN language with a more open syntax than it's predecessors , and great date math functions built in, and still manages to be backwards compatible to RPGII..OKAY SO I LOVE THE PLATFORM AND THE LANGUAGE SO :P ) so I *know* that they're compliant as they all use ISO dates. Also if they were going to fail, they'd have failed during our Y2K testing.

The problems I'm facing with embedded systems however are staggering. I've got several alarm systems that cross communicate with the door control and fire supression systems at the county jail, and (of course) one of the manufacturers has gone out of business. The county had a hard time deciding whether to replace or attempt to remediate. They opted to replace. But there's more going on at large at the county level. The time and attendance system for all county employees depends on a low level interface with electronic time clocks. Not to mention HVAC and door access systems at the Courthouse, Nursing Home, and other county facilities.

But the biggest problem I see with EDI is the supply chain. The 'Just in Time' supply paradigm is used by many retail outlets. Most food stores only have 72 hours of stock on hand, and maybe another 2 days worth in their regional warehouses (those chains that haven't done away with that distribution method). When supplies run low, their automated point of sale inventory system sends a signal to their warehouse, the warehouse to the distributor, and the distributor to the supplier, with a probable jump here or there to some mom&pop trucking company. If we loose power, it'll be a moot point, I expect a run on the supermarkets. But if we don't loose power but ONE LINK in the supply chain fails to process information correctly, there will be shortages and problems.

As far as embedded systems go, anyone remediating their communications with other systems have a really tough time. If they're lucky, the specifications of the interface are documented and available. Otherwise.........you get the picture.

And of course the IEEE would not point too obvious a finger at the embeddeds. That's what their members have been building and programming over the years. :-)

-- Prince Etrigan (etrigan@rpg.net), November 24, 1999.


Hey RPG, IBM showed up 2 days ago to uprade service director program on my As/400 (it was at 3.6). I thought we were already good to go. I upgraded the OS from 3.7 to 4.3 a couple of months ago. They just showed up out of the BLUE with the upgrade. The dude said the system may have crashed at the end of the year with out it. I don't even want to think about the 2 links I have with 2 state systems , It's all out of my hands. I run an AS/400 network, 50/50 PCs and terminals for a county court and juvenile detenion center. County gov's are so fu**ed up you can never get anything done. Testing ? whats that. All the "experts" think it will all be fine. I just do the day to day stuff, keep it all alive (150 + devices) .

-- Jeff (jstatzer@michiana.org), November 24, 1999.


Jeff --

I know what you mean about counties and testing. We got 3 days - count 'em - just THREE - for testing our Treas/Aud/Assr systems at a remote site. Got back and installed the *tested* Y2K revisions, and first thing out of the box, ALL tax parcels in the county were reset to tax year 1BC. Seems the system set the Assesment year to "00", and calculated the tax year by subtracting 1. Presto -- 1BC (-1). "That's 2,000 years of back taxes you owe sir. Sorry, computers don't lie!" Of course we got it fixed in a day or two, but such is life in a rural county.

Just a country programmer with 65 chickens and a generator

-- RPGman (tripix@olypen.com), November 24, 1999.


RPGman & Jeff - Your personal insights as to the scope of the problem are extremely valuable. Thanks for taking the time to post them.

-- Guy Daley (guydaley@bwn.net), November 25, 1999.

Moderation questions? read the FAQ