A VERY SERIOUS SITUATION: CORRUPT DATA

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

This is the thing that I am concerned about the most as with all my other posts about y2k. Corrupt data is not the kind of invalid corrupt data or garbage attached to a record only or a bad date. The data parameters (format) won't look corrupt it will look perfectly normal, the file itself may look normal, but the calculations of the data or the values itself will be wrong. But it will be very hard to tell the difference. Going to my bank the other day and seeing 4 out of 6 calculators all playing up and malfunctioning gives me some idea what could happen to the world's data structure on a larger scale. Once this kind of corrupt data sharing begins with the compliant or non compliant systems, (LIKE THE TRANSACTIONS OF THE BANK BETWEEN ALL THESE SYSTEMS) it will start trashing everybody's databases BEYOND REPAIR. You can screen out the bad data, and get the programs to reject it, but the corrupt calculations of that data will still remain even if you just filter out the bad or incorrect dates or garbage, it won't filter out calculations, for the calculations that created the data would be corrupt, so in turn this will through these thousands of transactions made daily end up corrupting the databases. Now I can't get plainer than this. There is the hidden danger of insidious data corruption that the eye cannot see to look out for, not just what the eye can see.

-- Brent Nichols (b-nichol@ihug.co.nz), January 05, 2000

Answers

skookum analysis. Y2K is, was, and will be a data problem, not, as it seems the world believes, a date problem.

-- sql guy (sql.guy@puget.sound), January 05, 2000.

I have been working on databases since the early 80's and what you are saying about corrupt data you cannot see it ridiculous. Data is data in the back end. How you display it might be different but what is stored in those fields is what there is -- nothing more. Why don't you stop making everyone afraid when you don't know what the hell you are talking about.

-- D Falcon (insider@idream.com), January 05, 2000.

It's only Brent, making his daily attempt to talk the sky into falling. Brent means well.

-- Flint (flintc@mindspring.com), January 05, 2000.

Banks balance nightly right down to the branch. Am wondering how such data corruption could possibly go un-noticed and un-repaired.

Some of the most complex banking conversions and mergers in history have taken place over the past several years and those worked after shaking out countless numbers of defects. Why wouldn't a date bug be able to be identified fairly quickly repaired....?

So long as standard accounting systems and practices are kept and done, even in non-financial institutions, how could such data corruption go un-noticed and un-repaired....?

-- KatInSeattle (YouC@ntSpamMe.com), January 05, 2000.


If the date data is not in the data files as 4 digit years (plus month and date -- e.g., YYYYMMDD), it is NOT compliant. If the program code does not read, write, and operate SOLELY on dates as/with 4 digit years, it is NOT compliant. "Windowing" and other ASSUMPTIONS as to century (19xx or 20xx) ANYWHERE (data or software) means data and/or software (as applicable) is NOT compliant. If NOT compliant, you are going to get bit in the ass in mysterious ways. This could be, granted, minor. More likely, major. Unless you consider hand examining and editing corrupt and POSSIBLY corrupt data as minor.

Ignorant pollies, shut up; you are demonstrating your ignorance. The fat lady has not even yet begun to sing.

-- A (A@AisA.com), January 05, 2000.



Flint; I think sql guy was sort of agreeing with Brent. I find myself thinking more about all the systems that have been developed over the years by accountants, lawyers, EE's, and even Computer Scientists that were set in place long ago where the programmer did something like use the date as the name of a file or the index into a record. You know those many spread sheets that some accountant wrote back in the days of VisiCalc (you do remember visicalc don't you)? I am talking about the little systems that are so critical to small businesses. The little guys did not remediate. Many of the little guys thought just upgrading their MS Office package could repair any problems, etc... etc... Many little guys don't even do regular backups as they should. Shoot! I have worked in educational institutions where they did not do regular backups as they should have done.

I have been called in on a number of instances over the years to work on such "little" projects where a change in the tax laws or a change in the reporting periods required a change in a spread sheet or small database. Those folks called because they did not have the inhouse to make even the simplest changes to the software the used so frequently.

Some technogeek wrote it an moved along to a better job. Some of these little systems are going to break, some will break in most interesting and novel ways. Some will break and the break will not even be apparent for months, probably beyond point that any backup will have anything but corrupt data.

What impact will that have? Your guess is as good as mine and I mean that. Probably in most cases the impact will be limited to a call to some guy like me or you who will go in and take a look and say something like, "Do you have a backup of your data?" or "Hmmm, If I can repair the flat-file database this system uses, I MIGHT be able to debug the application." "It won't be more than oh, perhaps 200 man hours total, if I am lucky...". Flint, some of those little guys can't afford me for 200 hours even if I give them a superb rate. This sort of thing is going to hurt them badly. Again how badly is anyone's guess.

You have spoken a lot of words of wisdom on this forum, stop for a monent and consider what I have written here and give me your honest opinion. I would like to hear it.

I am not worried about this. There is NOTHING I can do about it but wait and see. I don't feel badly because I told the people with whom I spoke that there could be problems. I really hope they made the backups, the way I recommended. That is the first thing they must have before I can start fixing that old flatfile database... You know the DBaseII stuff from the DOS days... etc... etc...

There was a heck of a lot of FOF in the small business community. Should be a bumper year for anyone that can fix those tiny apps, if they are willing to work real hard for small pay. Know many engineers or scientists that fit that criteria?

-- Michael Erskine (Osiris@urbanna.net), January 05, 2000.


Michael:

I expect a whole lot of what you describe. For companies with small resources, I don't see how it can be avoided. It'll probably kill some percentage of them. I wouldn't be surprised if the bankruptcy rate among small businesses were 10% higher this year than in a normal year (and startups to fill the holes higher as well, of course).

And I expect geeks will be putting in very long hours troubleshooting, not always successfully. GDP might even grow more slowly than expected as a result. In some cases, people two steps removed from the immediate problems might even notice! But I again caution you against collecting all of your data about community health from inside the emergency ward.

-- Flint (flintc@mindspring.com), January 05, 2000.


How the date is stored is not as important as how the software processes it or interprets it. Guess what? 2 digit ccyy can continue to work in many systems with the right logic wrapped around it.

Users know when their systems are out of balance or not working, for the most part. Cash registors must balance. Payables and recievables must balance. Inventory must be accounted for. Goods sold must = cash taken in. Checks written must = Bills payed. Etc. Etc. When they don't, PEOPLE act.

And there are regulatory agencies financial institutions must report to and answer to with regard to balancing...

...still tapping my fingers wondering how you guys believe all this corrupt data will just go un-noticed until it's "too late"...

not a polly here - am a pessioptimist. ;)

-- KatInSeattle (YouC@ntSpamMe.com), January 05, 2000.


It's only Brent, making his daily attempt to talk the sky into falling. Brent means well.

That was sweet of you, Flint.

-- semper paratus (I'm@serious.really), January 05, 2000.


Flint; exceptionally well said sir. Thank you. I hope you are completely correct.

-- Michael Erskine (Osiris@urbanna.net), January 05, 2000.


It does seem logical that errors will be detected by users, customers, etc. But then someone has to figure out what is causing the errors and fix the problems. Maybe it can be fixed in a few days and maybe it will take longer. What about the date that has already been corrupted? How will errors be found and corrected? While all of this is going on, the company could lose a lot if its business.

-- Dave (dannco@hotmail.com), January 05, 2000.

You would be surprised how many people do not pay attention due to to heavy of work load or just don't care. I have seen spread sheets, reports and accts. payable that are never proofed even for typos (it drives these same people crazy when I take the time to check my work).

I do not believe for one minute that these same type people are checking their formulas to see if they are correct or compliant. They just do what they have to do to get it off their desks because while they have been working on it someone has just placed another foot high stack of work on their desk. Plus you would be surprised how many people if they do find a problem don't want to bring it up because they don't want the attention focused on them, they just do not want to make waves. They think at the end of the day "just glided through that day and whew another day without getting thrown down to the mat".

Most companies I have worked for the past few years are terribly understaffed and press to much work on their employees. In my estimation the upper management has brought it on themselves by cutting budgets to save a buck at the expense of quality. As we have all heard what goes around comes around. Now it it the time the companies are going to get sc@#@% royally because of the actions they have been taking for years.

Maybe the new companies, who are born out of the havoc, will have learned a lesson and understand there is a limit to what a person can do in a day, if the company wants quality work.

-- Obo (susanwater@excite.com), January 05, 2000.


I heard from an ic manager at compusa who is a friend of mine that he has to work late tonight and for god only knows how long to take inventory because their jda systems are running two transactions for every one that is processesed in a certain way. These are the things that can cause a business to screw up their ibnternal reports that no customners are ever going to see will get corrupted and screw them up from the inside out this intern will screw up the number of items ordered shipped etc. And therre would be little short of going out of business that these companies could do about it.

-- Jaden (masterjaden@hotmail.com), January 05, 2000.

Brent:

Your theory is correct and completely sound. However so far emperical evidence has shown the effect you propose to be non-existent. Data has been transfered for the past 5 days. When do you expect these things to show up? The only comment I would make, is lets wait until the payroll start getting transferred into the banks on Friday this week and next week. And when the govt cheques go out at the end of Jan (actuall may have to wait till Feb sometime as I heard that the cheques for Jan in US and Jan/Feb in Canada were printed in Dec.

Flint:

WRT to small business I speculate the following is closer to reality:

(From my post in Hyatt: Y2K will not be a one time event...

Do you know what the problem here is? Most of us doomers are a bunch of amateurs when it comes to using scientific methods to arrive at conclusions. Look guys, we used the "belive the experts blindly" approach and we bombed out, royally. So lets stop re-hashing the same old tired arguments without subjecting them to some scrutiny.

I'm a doomer and sliently hoped the world would be shaken at its foundations for a few weeks because of the "Different stmuli, Different response" theory. Best that could have happened was that the wester world would have been shaken out of its lust for more and more - a, la David Suzuki. Well it aint gonna happen. Ok, I can live with that and I've got enough variatey in my stockpile to keep me eating the way I used to for 6 months or so.

But now we are into figuring out where this beast called y2k went and hid and lets not predict this using the same flawed approach we used pre y2k. Lets remember those who don't study history are doomed to repeat it.

So from the post at the top, for example we have the following said:

During the conference, a representative of the U.S. Small Business Administration (SBA) said there are 24 million small businesses in this country, with "small business" being defined as any company with 500 employees or less. When we reach January 2000, according to the SBA, approximately 8 million of these small businesses will be fully ready for Y2K, 8 million will have done enough repair work to muddle through, and 8 million will have done nothing to address the problem and will be at risk of failure.

Ok lets pick this appart rather than swallow it whole blindly and choke on it again later. I submit the following:

The 8m that will be ready probably have between 20-500 employees and therefore use a serious number of IT systems such that failure would cause a sever problem which is why they have prepared.

The 8m that will muddle through probably have between 3-20 employees and again will be ok since they probably can resort to manual payrolls etc. and "muddle through" as was suggested.

The 8m that did nothing probably have between 1-2 employees and are home based businesses (they are included in these stats) that basically have a PC with the usual suite of e-mail etc. and can fix all that quickly if it fails, i.e. they will obviously be in FOF mode (easily running on manual mode until failuers are fixed), but I am willing to bet that not one of those "businesses" will go bankrupt because of y2k because the just don't have "mission critical" applications that affect hundreads or thousands of customers. Those that do have failuers (because not all 8m will) will just have a bit of trouble putting out reports, letters and the like for a few days till they fix the PC, software etc. or just buy another one and get on with life.

So lets not get all fired up about these "small businesses" causing massive unemployement etc.

Time for the doomers to wake up a bit and realize, it will not be death by a thousand knives, but sort of like being eaten alive by a thousand ducks. It sure going to feel bad while they try, but it just aint going to happen in the end.

-- Interested Spectator (is@the_ring.side), January 05, 2000.


I have been working on databases since the early 80's and what you are saying about corrupt data you cannot see it ridiculous.

That's when I started. Subtle corruption is the kind you can't see because it looks normal, though the numbers are wrong. It's like 1900 vs. #@$%

-- Slobby Don (slobbydon@hotmail.com), January 05, 2000.



Was it Chase that misplaced/lost $60 Billion in 1999 for several months, but apparently the books balanced?

-- snooze button (alarmclock_2000@yahoo.com), January 05, 2000.

KatInSeattle,
I will try to address your question with an example. Let's say I have an interest bearing account that's subject to daily compounding. For the first bank business day of the month of January, my bank computes my interest credit to be $9.65 (after first applying any applicable charges and credits). Each business day of that January, similar calculations take place, apparently without a hitch.

During the first week of February, statements for January are generated and mailed. During the second week of February, I receive my statement, notice that the interest credited to my account is $50 less than I expected, and seek to rectify the discrepancy with the bank. Turns out that 1,000 other customers were undercredited also, due to an error in the bank's interest calculating program.

This miscalculation did not cause anything to be "out of balance" in the bank's records, and the results, though wrong, might well appear perfectly normal. The only way to catch something like this is to do what the bank's computer is supposed to do, that is, start with the end of December balance and apply each day's charges and credits to the account, until you've calculated a balance for the end of January. (Actually, the calculations that would have been made thus far in February would also need to be adjusted.)

Moreover, I'd imagine that a mistake involving entities in different countries, and therefore having implications for international settlement, could have far more serious repercussions.

-- David L (bumpkin@dnet.net), January 05, 2000.


I've also been involved with databases since the mid 80's from a systems perspective. I have experience with what will corrupt a database and how they can be recovered. The databases I am responsible for are on multiple operating systems. One of our databases is huge (terabytes) and supports hundreds of active users.

For any shop that has critical data, there are plans in place so no database is able to be corrupted beyond repair. I've had to recover some pretty trashed databases and was able to. This is for the type of corruption that will crash your database. The type of corruption that yields invalid values but doesn't crash the databases is another issue.

For the corruption that is spoken of on this thread ... valid data with invalid values.. I've also seen a system detect and recover from this problem.

I am not an accountant and I have no accounting background but I know the programs that run on our databases have checks and balances in them to detect valid data with invalid values. How they do this I haven't a clue, but I've seen it done.

Our main accounting system was recently migrated from one system to another. Very large and complex job for the accountants and their programmers. They had tested and retested the new system, running in parellel with the old, to ensure all was ok. Things looked fine. Cutover to new system beginning in a new fiscal year. Ditch old system.

New system put into production. About a month before the close of our fiscal year, it was discovered that invalid values were being placed into the accounting records. This (storing invalid values) had been going on for several months. It wasn't just a matter of one field with incorrect values, the problem had multiplied because of incorrect sums posted on other records. Since our shop allows users to download accounting data for their own use, the 'corrupt' data had multiplied and had invaded our users' spreadsheets, etc.. It was a mess.

The problem was detected by a series of audit jobs our accountants normally run. It was not an easy problem to fix (I'd rather recover a trashed database), but it was fixed.

There is always the chance for invalid values to be placed into accounting data when a program is being updated. This was one of the reasons the audit programs were put into place. I'm sure the Chase Bank has similar programs in place and that is how their error was caught.

So, I wouldn't worry about data with invalid values populating databases and causing all our financial systems to be useless. Most shops that have electronic financial data also have programs designed to find such things.I know our accountants analyze the results they get from their audit programs. It's not just trusting a program to detect problems. It's knowledgeable accounting people analyzing the output from the audits and starting to investigate.

I'm not stating that all invalid values will be detected in every system. But, the likelihood of invalid data being so widespread that our financial systems are at risk is very low. There are checks put in to detect this.

The important point about what happened at Chase is that it was detected.

-- Chris Josephson (chrisj62954@aol.com), January 06, 2000.


Chris; well spoken. Still there are those nagging little systems mentioned in the exchange between Flint and I on this thread and addressed by 'Interested'. Those little folks make up a pretty good piece of the economy. I will not be surprised if the IRS decides to cut them some extra time this year once they figure out that there are going to be accounting problems in small businesses. That seems to be the easiest way to allow the really small ones to get things sorted out without having to hire on new help or just go belly up.

There is a report of Wachovia calling the loan on a local restaurant because they were ONCE (not currently) late on a payment. This was on the local news last night. This is apparently an isolated incident though the details are very sketchy. I mention it here because that accounting software MUST be running for a small business to make it's payments to creditors ON TIME. Unfortunatly creditors don't usually care whether you are going to have it fixed in a month or two, they just want their payments on time. What amazes me about this particular instance is that Wachovia had a clause in the contract that stated if the holder was EVER late in the lifetime of the note, Wachovia was subsequently entitled to call the full balance due whenever they wanted. The date they chose was 1-1-00... put those people right out of business. Now, how many people have such notes? What percentage of small businesses are going to be late on atleast one payment because of software problems related to CDC? Both questions can only have speculative answers, but are never the less, interesting questions.

Interested; also well said. I mostly agree with you and Flint. In particular your rational seems reasonable. I don't "expect" a disaster, I do expect this could (note the hedge here please) trigger recession in conjunction with other factors which may have occured because of Y2K but are not attribuitable to Y2K, such as too much liquidity. Could there be an actual market crash? Sure. Will there be? No idea at all, I can't even predict the precise time I will need to get up from this chair and head for the bathroom.

-- Michael Erskine (Osiris@urbanna.net), January 06, 2000.


Michael:

Agreed. I suspect there will be problems (? how many) like you pointed out. Wasn't trying to imply there would be no problems. Was more responding to some of the over-statements I've seen that imply we have a huge problem that will cause some financial crisis on a national scale.

People who don't work with databases can be unaware of what goes on behind the scenes to ensure the data in many companies is reliable. Especially if that person's exposure has been in small or personal databases that may not have the safeguards found in enterprise-level systems. Heck, even the database programmers in my shop don't really know all that's done to ensure the data is reliable and available.

I would hope, with all the publicity Y2K received, there would be an understanding when financial mistakes are discovered. It could take a while for the smaller problems to be found.

Hopefully the 'Y2K backlash' (there was no Y2K problem) won't stop people from being dilligent in looking for data that may be invalid.

-- Chris Josephson (chrisj62954@aol.com), January 07, 2000.


Moderation questions? read the FAQ