1/30/2001 Summary of an Engineer's Observations Regarding the Status of Ongoing Y2K-Related Embedded Systems and Complex Integrated Systems Problems
Compiled by Paula Gordon (With a minor revision made 2/1/2001)
Introduction
Problems of the sort that were predicted prior to the January 1, 2000 rollover have been occurring in a wide range of sectors. The following summary of observations of an engineer provides abundant clues concerning the possible causes of problems that are evident in the energy sector and in other sectors as well. Preface:I am attaching a summary of observations that I have drafted. The summary is based on observations that an engineer has shared with me. I am sharing these summarized observations for several reasons:
1) because of the relative absence of first hand accounts concerning what is actually going on regarding Y2K-related embedded systems and complex integrated systems problems;
2) because I have heard some similar off the record accounts from other engineers; and
3) because I feel that right now such off the record observations provide the best information and roadmap to further inquiry that we have.
Perhaps, those who are in a position to do so, will come forward, at least off the record, and help enlighten the public and those in positions of public and private sector trust and responsibility concerning the significant role that Y2K-related embedded systems and complex integrated systems problems are having in a variety of sectors, including the energy sector.
********************************************************************************************************************************************
During the last week of January 2001, I received some information from a seasoned engineer who has been working "on the frontlines". The identify of the engineer cannot be disclosed since the individual's job security could be jeopardized.
The individual shared information concerning the many Y2K-related problems that he is continuing to see. (I have not met the engineer in person and do not know his or her real name and will refer to "him" as "he" in this summary.)
Also, rather than quote the individual directly, I am summarizing most of the information that he shared with me.
~ Several of the companies that he has worked with have had extremely serious data corruption problems. After much effort and temporary successes in dealing with these problems, the data becomes corrupted again.
~ With respect to the grid, he feels certain that the energy crisis will become increasingly apparent this summer. In his view there have been large numbers of failures involving energy systems. In these instances, he says that workarounds are often not possible. He notes that turning clocks back and going to manual have resulted in some cascading failures and time delays.
~ He notes increasing reports of problems with dirty power and low power and instances of involving the total failure of electrical equipment.
~ He also talks about what he feels is a direct correlation between solar storms and hardware failures.
~ He says that those working "on the frontlines" are being threatened with the loss of their jobs if they speak up about what they know.
~ I had told him that it was my sense that people at the top of private sector organizations do not seem to comprehend the extent of their Y2K-related embedded systems and complex integrated system problems. He said that of the persons he comes across, less than 20% of those who work with complex systems understand the systems and keep up with changes and that only a small percent is able to address problems effectively. The others don't really understand what is going wrong with their systems.
~ I asked him how large a role he thought Y2K-related embedded systems and complex integrated systems problems were currently playing in the evolving energy crisis. He said that he estimated that 70% of the failures involving the energy sector, and communications (among others) are directly the result of Y2K. He estimated that 20% of the failures could be due to human error on the part of those trying to deal with the problems. He said that those individuals often only have enough ability to deal with normal activities and that they have insufficient understanding to deal with anything that departs from the norm. He estimates that the other 10% of the problems is owing to normal hardware failure, user problems, and environmental issues.
~ He said that manual override and date resetting have been used when automated production systems and SCADA systems have failed. He said that it is not uncommon when he is replacing a system module to be told by the client that he has to put in an old date or the application will not run. He added that many of these applications are old and that large networks over the past decade can be composed of a mix of upgrades, networks, and applications that are out of sync. Owing to these problems, he estimates that the country is running at 65% to 70% of last year's production rates on the average.
~ I asked him about problems in all of the high hazard sectors: oil rigs, refineries, oil and gas pipe lines, nuclear power plants, nuclear reactors, chemical plants, hazardous material facilities and sites, electric power plants, water purification plants, waste treatment plants, trains, planes. He responded that most of these have fixed what they could; fixed the rest on failure when possible; or, if the expertise is missing, attempted to make the failing system work manually. In situations where a system is run 24 by 7 and where there is an apparent problem, he says that there is only a narrow window of time during which the system can be analyzed and repaired. Sometimes when there is an apparent problem, but where no hard errors have occurred, he has been asked to replace hardware. When new hardware does not fix the problem, going to partial manual override becomes the only remaining option. He also noted that in many networked environments, date/time is sent in packets and when there are systems broadcasting an old date along with current dates, the data can be corrupted or miscalculated.
~ He said that he has not found anyone who is willing to talk about what is happening, even off the record. He said that some of his more aware customers are asking him what he is seeing and asking questions about the power crisis. He thinks that they are beginning to catch on.
~ I asked him if he knew of any cases involving high hazard sectors where the problems are being publicly recognized AND linked to Y2K? He said that Y2K is never mentioned in explanations as a cause of problems. Instead "silly" explanations are offered and most people take these explanations as fact.
~ I asked him what his prognosis was for nuclear power plants. He said that he was told prior to the rollover by someone in a position to know that in instances that his information source knew about, clocks were turned back where there was a possibility of potential problems and failures. He said that this only works for a time as the interconnectness of these system runs too fast for individuals to keep them going. In his view, the production task has become very costly negating most, if not all profit. In addition mechanical/electronic failures are extremely costly. He said that he felt that many nuclear power plants were running well below capacity due to the failures and owing to manual operations. He feels that they do not seem to be making much progress getting back to normal and that in the end those plants will become too expensive to run.
~ I said that I have been hearing about shortages in the pharmaceutical industry and asked him if he thought this might be related to problems with manufacturing processes. He said that there are manufacturing problems and that too many bugs have slowed manufacturing processes. He added that there is a major shortage of computer components and that the parts that are available are often parts that have been put back in stock even though they do not work. He said he has found the same to be the case when it comes to other technology companies and parts vendors.
~ Regarding health care system problems, he said that they are having all kinds of issues, including claims that are getting rejected for no valid reason, accounts that are coming up blank, or billing where charges and services are being doubled.
~ Regarding air travel, he said that air travel is having its share of Y2K issues. He also feels that solar storms are having an impact on air travel and that Y2K coupled with solar storms have triggered many of the problems that have been occurring.
~ I asked him what he thought about the possibility that manhole cover explosions might be caused by irregularities in transmission. He said that the manhole issue is a very interesting one and that he feels that it is due to electrical power cables overheating and creating a gas that results in an explosion. He thinks that this is probably due to the use of manual power overrides.
~ He said that every time there are major solar flares, he notes an increase in CPU, memory and disk drive failures. He notes that the incidence of failing modules is very high owing to their density, a factor that makes them more sensitive to the effects of solar storms.
~ I asked him if he knew of any cases where problems involving data degradation were being publicly recognized AND linked to Y2K. He said that not one company is going public. The usual explanation is that the company is having "computer problems" and that "the system is new".