Oil & Y2K A Final Evaluation

Part 2 -- A Brief Review of the Facts

Note: This is not a comprehensive undertaking, but a brief review for folks new to the issueWhen I posted my first comments on the oil situation I did so as a 'report' because it was a little too lengthy to just insert into someone else's thread. Since then, people have asked me to provide updates. I have done two other updates. This one is my final update before the rollover. What is to follow is premised primarily upon the embedded systems issues within the oil sector. We have some numbers upon which to calculate potentials. We have testing results, albeit very limited test results from which to draw conclusions. This being said, we can only state that what I am presenting is a "Shot in the dark" guesstimate based upon scant numerical information that is publicly known.

We will attempt to extrapolate from this some probability factors for what may ensue during and after rollover. In some ways it is like predicting the weather, but we're focused on the Oil sector of Y2K only. It is based upon somewhat crude information because it is all we have to work with. What we present may be too optimistic or too pessimistic but nobody out there can produce anything better at this point or we'd have seen it. These guesstimates should therefore NOT be taken as "gospel-truth" but neither should they be disregarded.

Our attempt here is to be biased in favor of optimism in what could be viewed upon as a myriad of extremely pessimistic data points. We're not out to create hysteria or panic. Just the opposite. We do feel that the data justifies a high level of concern that should underscore the need for being prepared for problems that could cascade beyond the oil industry and spill over into other sectors of society.

Before we look at predictions, we need to review what we do know in various sectors of the oil industry. Those sectors we will look at include:

A. Oil well -- crude oil production (both domestic and foreign imports)

B. Oil transportation -- pipelines, ocean tankers and port docking facilities

C. Oil Refining -- refineries

D. Retail Distribution -- Your local gasoline station

NOTE: I do not have sufficient quantities of sources for me to confidently address the natural gas issues. There may or may not be cross-applications to consider but I will not deal with these. I will only deal with what I know with some level of strong likelihood or certainty. After reviewing these above areas, I will then present an evaluation of the data that suggests what outcomes may transpire and the percentage of likelihood.

BACKGROUND -- Reviewing the oil data

A. OIL WELLS and CRUDE OIL PRODUCTION:

Oil wells can be divided into 2 categories.

1. Large oil wells. These are loaded with automated computer systems that are composed of what are called "large-scale embedded systems." This automation reduces the need for human interaction and thus lowers labor costs. The automated systems are cost-effective because the well pumps enough oil to make it financially affordable. These types of wells account for nearly 80% of US domestic oil production and supplies about 36% of US needs.

2. Small oil wells. These are sometimes referred to as "stripper wells." These wells produce small amounts of oil. Stripper wells (generally) are found in older oil fields that were exploited long ago. It is not profitable for an operating company to install embedded systems onto these wells. These types of wells account for nearly 20% of US domestic oil production and supplies about 9% of US needs.

In regards to Y2K, "stripper wells" are and always have been compliant. They contain no automation. Large oil wells may have a compliance problem due to the large quantities of integrated computer chips installed within various embedded systems or Large-scale embedded systems (called LSES). The average large oil well will likely contain on average around 10,000 'chips' placed in approximately 50 to 100 Large Scale-embedded systems. Private industry assessments from preliminary testing indicated that 1 in 3 LSES is likely to experience a systems shutdown upon rollover. This does NOT apply to newer large wells that were developed within the last 2 years generally speaking.

Thus we have the potential risk of losing 36% or more than one-third of US supply needs if all of these wells fail. Virtually every large well runs this risk. Newer wells of the last 2-3 years make up a very small amount of the large oil wells and these are estimated to provide (at best estimate) less than 1% of that 36% figure. This leaves us with 35% being "at risk" for failures if not remediated.

The problems with remediation are multiple, according to my sources. The primary problem is testing for problems. Most oil companies do not know how many potential chip problems may be in a given well because each well was customized to unique situations present at that location. Many installations were not well documented or else those documents are now long gone. Many times wells are bought or swapped and over a 20 year period may have several different oil companies operating that well. Papers and documents tend to get lost in those situations.

Testing also necessitates that the wells be shut down. This is a serious problem in that in order to shut down a well the test results may be so severe that the well is permanently lost. In addition, testing a well could take too much time to do a realistic and effective job. Another problem is that when testing is performed and problems found, often times the chip cannot be easily accessed without destroying the sealed system in which it is placed. The system was not supposed to be repaired but rather replaced in toto should it ever fail before the well runs dry. In many cases, it was expected that the well would go down before the embedded systems. It is usually not the embedded system that goes out before the well does.

These systems are usually customized and are not something bought off of a shelf and placed directly into the operation. Most often they are built from scratch to specific order or modified from assembly line production. A customized system can take 3 to 4 months to build and install and test-out before placing online. Many of these types of systems have been referred to as inaccessible systems, because for all practical purposes it is inaccessible because they are sealed systems. To access it is to destroy the seal and because it is in a sealed system that cannot be resealed once it is tampered with. Also, within it, parts are coated for protection and cannot be identified because of the protective coating. It is a part of weatherproofing for the elements directly onsite. This doesn't necessarily apply to all the systems on a well, but it does to many. Some systems are also difficult to reach and may require disassembly of machinery to reach them, just like it may require disassembling part of your car's engine to reach certain computerized sensors that assist in controlling vital aspects of (say for instance) fuel injection. This is true also with an oil well.

Another problem in remediation is that some computer chips cannot be replace even if not in a sealed system because there are no longer replacement parts for that chip. Thus an entirely new system must be designed and constructed. Again we're looking at perhaps a 3 to 4-month lag time. Also, there are only so many folks qualified to do the designing and building of these systems. It is not likely replacing a transistor or IC chip in a VCR or TV set, nor like building a VCR or TV on an assembly line. There are no assembly lines for most of these systems being hung on oil wells.

So we have a whole series of problems involved in trying to identify and fix failures. We have a problem in lost revenue during a well's downtime. We have a threat of losing an oil well in testing. We have a problem in getting a chip or system replaced in a quick and timely manner.

We have a problem in terms of the rate of failures on the LSES on large oil wells. The rate of failures in LSES systems is 1 in 3 (average) on a well. There have been instances of 1 in 2 or half of the systems reporting failures. This means that anywhere from 15 to 50 systems may have severe problems that can shut down the well. Generally speaking there are at least 1 to 5 critical systems that can be expected to shut down that well due to date rollover problems. Many of the other system may fail but will not necessarily shut down production. The bottom line though, is that it is expected that at least one to five of the 15 to 50 systems are expected to have critical failure shutdowns. These are not very good odds.

B. Oil transportation -- Pipelines, Ocean tankers, and port docking facilities.

Pipelines are a primary means of transport for domestic crude oil. It provides a means of moving the crude oil from the wells to refineries where it is refined into gasoline, diesel fuel, kerosene, lubricants, etc. In ye olden days of yore, (about 20 years ago) pipelines began to undergo automation improvements designed to eliminate skilled laborers and high payroll costs. Automation was computerized utilizing the SCADA embedded systems. The SCADA systems in pipelines are not exactly the same as found in say the electric utilities. Pipeline SCADA systems are more complicated in applications and do indeed involve time-date issues at periodic checkpoints on the pipelines. Testing on an "end-to-end" basis could not be done without shutting down the entire system. This was not done. There was very little testing involved except for "type-testing." Based upon this limited testing, the industry has concluded that there is no problem.

Embedded systems experts in the oil business however, are privately not nearly so confident as the corporate PR news releases indicate. Frankly, from what I've been able to discover, it is truly an unknown status and a real wild card issue in assessing the likely impacts of rollover because there is so little that is truly known. In order to know, testing would have necessitated in disrupting the flow of oil in a major way that would have had severe economic impacts upon refining of crude oil products and thus affecting the entire economy by limiting refined supplies.

Ocean Tankers are also loaded with embedded systems in order to reduce labor costs by reducing the number of crewmembers needed to run the ship during voyage. Reports I've seen indicate as many as 150 LSES aboard and control a number of critical elements aboard the ship. Information regarding testing results is rather scarce and incomplete. Suffice it to say that reports I've read indicate that at least one or more systems are expected to experience problems if left unremediated. This has been a real concern for the US Coast Guard. The USGS has issued new rules mandating that ships that are not compliant will not be allowed to enter ports during or critical passageways during the rollover. I have not been able to gain enough data to assess the mathematical impact of this.

Here is a recent report on the US Coast Guard's position regarding this issue. From Federal Computer Week: http://www.fcw.com/pubs/fcw/1998/1207/fcw-newscoast-12-7-98.html Snippet:

" sAN DIEGO -- Year 2000 date-code problems in oil tankers could lead to an oil spill in a U.S. port or slow the flow of imported oil to U.S. communities, some of which have "only a three- or four-day supply of oil,'' the Coast Guard's chief information officer said last week."

Speaking here at the Navy Connecting Technology Conference, Coast Guard CIO Rear Adm. George Naccara said concerns about computer chips that control oil pumps on vessels and in on-shore storage facilities could force the Coast Guard to issue a directive telling tankers "not to offload cargo on Dec. 31, 1999." The Coast Guard can issue an order against oil offloading from tankers under its mandate to ensure marine safety in U.S. coastal waters and ports. The Coast Guard also is worried about how the Year 2000 problem will affect the marine transportation industry, especially the 7,000-plus foreign-flagged vessels, which make more than 80,000 visits a year to U.S. ports. Over the years, these ships have been automated with information systems that have allowed ship owners to cut costs by reducing the size of crews. Today, even the largest tankers can operate with only a crew of 10.

Naccara said the embedded-chip problems in some ships could come close to halting them dead in the water. "Some [vessels] may have problems crossing the ocean,'' which also could put a crimp in the nation's oil supply.

Here is a link to another Maritime website that spells out more details of problems with embedded system aboard ships:

http://www.ship2000.com/Ship2000.nsf/5a8ba7baee61c43c802565df004ad3ff/29a84b3de62b2a0e802566b300416c04?OpenDocument

Port Docking Facilities are also involved LSES equipment for loading and unloading oil. There are reports and indications that many facilities are not compliant and will not be compliant for rollover. Again, as with the tanker issue, I have no first-hand eyewitness sources indicating any information one way or the other on this. Based upon what I've read though, and the US Coast Guard information, there are apparently many port facilities around the world that are not ready. Reportedly, Venezuela and Saudi Arabia are having problems in gaining compliance, but this is subject to debate.

In the above-cited report in Federal Computer Week they also had this to say about on-shore problems

" The Year 2000 marine infrastructure problems extend beyond oil tankers, Naccara said, to items such as the giant cranes that offload containers from cargo ships. The Coast Guard also has started to develop contingency plans to mitigate problems with critical infrastructure systems, such as the electrical power grid.

Naccara said the Coast Guard has an ongoing "broad outreach effort'' to provide Year 2000 information to every segment of the marine community, including briefing the International Maritime Organization and distributing Year 2000 brochures to foreign-flagged tankers and recreational boats. Despite these efforts, Naccara said, "I have no doubt we have great exposure [in the marine transportation environment] due to the weakness of embedded chips."

End Snippet.

As you can see, there are some serious concerns raised by the USCG in this regard to oil tankers and shore loading/unloading facilities. One aspect that is scary that I didn't post was the comment from the International Maritime Organization of London spokeswoman who claimed that this issue of Y2K compliance was unknown to the safety division of that organization!! Hello????? See the article and also note the segments dealing with Chevron's exposure of its 35 tankers which were undergoing testing during December.

C. Refineries

Refineries in the US are primarily old facilities. Many date to the early part of the century and have been periodically remodeled many-times over. The larger refineries have large elements of LSES within them. This aspect of LSES in refineries is a closely guarded secret. It is one that the oil companies have not seen fit to publicly share. My insider sources tell me that most refineries with significant quantities of LSES are at severe risk. Specific refineries have been mentioned to me but sources have asked me not to reveal them because of possible repercussions to those sources who fear that with so few knowing the status the leak could/would easily be traced back to them.

Not all refineries are loaded down with LSES. Some have very few of these. Very small refineries do not have them in part because of the extensive costs and because they are owned by smaller independent oil companies. The Major Oil companies are the ones that have loaded down their refineries with embedded systems (LSES) in order to reduce expensive high-wage (union) jobs. Some refineries can be run with only a handful of men if heavily LSES'ed. There are two of the Major oil companies that reportedly have very limited exposure to the LSES problem and yet maintain larger to moderate sized refineries. These two oil companies are Marathon Oil Company and Ashland Oil Company. Both are subsidiaries of USX. (the old US Steel Corp.).

Most of the other majors BP/Amoco/Arco, Chevron, Texaco, Exxon-Mobil, Shell, Conoco, Enron, Gulf, have extensively equipped their refineries with embedded systems. However, again, not all refineries are equally loaded down to the same degree. Another compounding problem is that many refineries in the last year or so have been sold to independent or foreign companies. The major oil companies used to compose 70% of refining capacity. They are now down to 54% at last report. Independents now account for 23% of refining capacity. Mergers and acquisitions since these reports of a year ago probably have further altered those figures dramatically lower. Many of the independent companies have not had the deep financial pockets to perform remediation like the major oil companies. This may result in compounding the problems of Y2K remediation.

What do we know about Y2K remediation in refining? The oil industry is not providing details anymore about its progress. We can however refer back to this classic webpage for an interesting article on one Y2K remediation company's work with a major oil company. This article describes the remediation company's efforts and those results at the 18 refineries owned by the Client Company. We'll post below that some relevant quotes pertinent to our discussion.

http://www.prepare4y2k.com/embed1.htm

Begin article quotes:

"The results for the manufacturing systems were that 11% of the hardware, 14% of the software and 23% of the custom code was non-compliant. 2% of the suppliers were no longer in business and 12% of the suppliers would not supply an upgrade except through purchase of new models. "For the quality systems, 24% of the hardware, 29% of the software and 29% of the custom code was found to be non-compliant. All suppliers were still in business but 16% would not provide upgrades except through new models. The risk of failure was put at 60%.

"For the co-generation plant's results, 19% of the hardware, 36% of the software, and 24% of the custom code wasn't compliant. 13% of the products were no longer made. In this case, the energy management system tied into the local grid that was non-compliant and couldn't be upgraded. Some of the more interesting findings were that the catalytic cracker would fail and the refinery could no longer make gasoline. One of the more troublesome findings was that the analyzers would continue to work but would send erroneous data. The proprietary networks from the control systems to the analyzers would fail. The inventory and analysis would take 7 weeks and cost $122,000. The conversion for two units would take an estimated 15 weeks and cost $760,000.

"In Summary "In an overview of all these facilities, problems were found in the sensors & analyzers, lab equipment, programmable Control Systems, embedded systems, SCADA systems (Supervisory Control And Data Acquisition), distributed control systems, human-to-machine-interfaces (HMI's), networks, computers, third party applications, and Operating Systems. "Mr. Heerman also noted that they had found that some of the vendor compliance statements were incorrect. Some pieces of equipment the vendor had claimed to be compliant had failed. One piece of equipment successfully made the January 1, 2000 transition and was allowed to continue. Just over a month later, when checked again, the date on the equipment was January 34! This points out the need for continued testing, especially at the system level.

"The projected risk levels for failure of all the units of these companies was between 60% and 90% if the non-IT parts of the business were not found and fixed."

And another archived article confirms the premise:

http://www.cio.com/archive/090198_y2k_content.html

[snip] "Circuit boards aren't easily accessible," says Peter O'Farrell, a senior consultant with the Cutter Consortium, a research and consulting firm in Arlington, Mass. "You may have to completely dismantle the system to get at the circuit board. Then you can pull the chip off the board. But what if there's no new chip with the same footprint? Plus, you have all the static electricity issues, contamination issues and waterproofing issues. Messing around with boards is not much fun." Especially if they're located at the bottom of the North Sea. Ascent Logic Corp., a risk management technology firm in San Jose, Calif., has inspected Y2K projects at several oil companies with embedded chips on the sea floor that control wellheads. "Most of those chips will shut down on Jan. 1, 2000," says Ascent President and CEO Larry McArthur. "The question now is, Is there enough diving capacity and time remaining to get a significant portion of these replaced?" Be sure to order replacement parts and related services as soon as possible. "For every industry I've looked at, the order queues are filling up," says Roleigh Martin, a software engineer consultant in Edina, Minn. [end of quote]   

[snip]

"Many embedded systems are effectively "black boxes," purchased ready-made from a vendor without knowledge of the components or code that makes them work. "If we have a black box system controlling our HVAC system, no one within Amoco is likely to be intimately familiar with the contents of that box," says Hale. "As a result, you're very dependent on the supplier of that product for information that would inform you whether or not you have a Y2K problem."

[End of snippets]

My comments:

Now, the following is worth repeating again:

" One of the more troublesome findings was that the analyzers would continue to work but would send erroneous data. The proprietary networks from the control systems to the analyzers would fail. The inventory and analysis would take 7 weeks and cost $122,000. The conversion for two units would take an estimated 15 weeks and cost $760,000."

Did you catch that? "the analyzers would continue to work but send erroneous data." This tells me that some data will take a while to shut down systems. Others could be instantaneous. Note the length of time to repair this particular problem 15 weeks. That is nearly 4 months! Note also that (as I understand it) Problems like this could be hidden by instantaneous shutdowns that will mandate fixing before the other problems surface. So thatone item could shut things down instantlytake a long time to resolve and then have data corruption problems elsewhere surface perhaps 30 days after resumption of systemsthus shutting the systems down again for perhaps months.

So what are we saying? It is possible that refineries will have problems of perhaps of an immediate nature, and then again after repairs are made and systems resumedonly to see further data corruption (that had been halted) resumed without operator realizationonly to have further shutdowns days or weeks later.

In other wordschain reactions of problems in which we see a series of shutdowns in a chain that spreads out over time that could be hours, days or weeks apart. It could mean perhaps a half-year to a year of off again and on again shutdowns. In the case of refineries. You don't just restart a refinery by flipping a switch. It takes time to power up or rev-up a refinery, usually measured in days or weeks depending upon problems. This could be a real mess for refineries. Sometimes to clean up messes from a sudden shutdown can take perhaps weeks depending upon the severity of the problem and the extent of turnaround needs.

The stakes could easily result in extended losses of refined production levels across the oil refining industry.

D. Retail distribution -- your gasoline station

From what I've been able to uncover, this is one area where most of the industry is prepared and ready. Whether they are fully compliant remains to be seen. It is the one area where they faced the least amount of challenge, so I tend to think that most retail gas stations will be okayalthough perhaps their credit card systems may have problems that are perhaps related to difficulties with phone line access. Nevertheless, I am particularly optimistic in this particular area. This of course assumes that the power grid remains up and running.

-- R.C. (racambab@mailcity.com), December 27, 1999

Answers

Good grief!

-- Ocotillo (peeling@out.===), December 27, 1999.

So far, this is quite alarming. However, i have one question. You stated that Large wells comprise 80% of domestic oil production, and supply 36% of our oil needs. You then say that as much as 1/3 of these could fail. But then you say this will cause us to lose over 1/3 of our oil supply needs. Does this mean you are extrapolating this failure rate to foreign oil? If 1/3 of our domestic large wells fail, we will lose 1/3 of 36% of our supply, or 10% total. Granted, this is still a problem at our current rate of comsumption, but not the major hit you claim. I realize that foreign oil will probably fail at an even higher rate, and thus your claim is, as you say, optimistic, but by not mentioning foreign oil your numbers may confuse people, and leave room for the polies to say "See! he's inflating the numbers to incite panic etc etc..."

Jes' Thinkin'...

-- Little Pig (littlepig@brickhouse.com), December 27, 1999.

Little Pig,

good point on the stats - see part 3 for the lowdown which address your concerns...

-- Andy (2000EOD@prodigy.net), December 27, 1999.

R.C.-Thank you for the time and effort involved in assembling this information and assembling it in a coherent format. Your effort is deeply appreciated.

-- Blew5M (gaf@mindspring.com), December 27, 1999.

Little Pig,

To attempt to bend over backwards in favor of optimism, I've referred to the 1/3rd aspect only for foreign sources that are rated "high" likelihood for problems by CIA to Senate or the latest DoE report dated 1st week of Dec 99. Because we don't know which wells and of course not all wells produce the same, we're assuming then that the 1/3rd of the wells translates out into 1/3rd of the supply...and that it is equally divided to all the nations that country exports to. Thus we're assuming the USA won't try to hog or steal extra oil from another nation counting on oil from that foreign country.

re; the numbers... you're correct on the 1/3 aspect of US large wells at 36% and that a third of that would be 10% of our USA total needs. You are correct. Yes it can get confusing. "Who's on first?" :-)

Actually a 14% loss is the most optimistic and that is still far worse than any other oil crisis we've seen, by about double or triple depending on whose numbers you take. Technically, the DoE figures would indicate that the 73/74 embargo crisis only dropped us about 2./2% as it only lasted for the last 3 months of 73 and the first 3 months of 74... irregardless of 2.2% 4.5% or 7%... a nearly 15% drop now would be far more devastating than 20 or 27 years ago. And I'm being the extreme optimist to show that BITR is simply completely illogical.

Thanks for the input. Got bricks??

-- R.C. (racambab@mailcity.com), December 27, 1999.

The assessment data you cite for 18 refineries of a major oil company comes from TAVA; I'm not sure, but I think the oil company was Chevron. At any rate, TAVA has had Chevron as a client.

That one out of three LSES in the oil industry is vulnerable to serious Y2K problems or failures is very much in line with the numbers I've seen during the past 18 months from a variety of reputable sources (Gartner, etc.) on LSES across the board. And the time to repair can be disconcerting: the International Energy Agency in its IEA Y2K Documents section of its website notes the case of one oil company LSES with Y2K problems that took six months to fix, from initial detection of the problem to final testing of the system. Folks can also find at the IEA site (www.iea.org) the Ascent Logic Corp. info that you present.

I don't know how widespread or truly major Y2K problems will turn out to be in the oil industry, but if there are serious, common, unremediated problems out there in LSES--DCS, SCADA, etc.--then a nasty time is indeed ahead.

-- Don Florence (dflorence@zianet.com), December 27, 1999.

Don,

Almost missed your post as most folks are posting on part 3 page.

Glad you posted and glad you brought up the name, Chevron. Yes, Chevron is one name that keeps coming up over and over and over again. I suspect that we've not heard the last of Chevron having problems. I think the 18 "plants" includes other processing units besides oil refineries producing gasoline/diesel/kerosene/heating oil. There are no doubt others referring to petrochemical processing for plastics and other by-products of oil. I'm not sure just how many oil refineries Chevron has. I've hunted for those numbers a couple of times and got sidetracked. You know how it is when you go surfing the web? I didn't realize Tava had Chevron as a client...but I suspected it was Chevron or Amoco. Thanks for the information. I may have to go out and do a little more surfing again.

-- R.C. (racambab@mailcity.com), December 28, 1999.