The Plan

The Plan

It has been an interesting couple of days in the US, in the world, with regards to the COVID-19 virus. I actually have two or three posts sitting idle, all still in the edit phase, all of which are likely to never see the web as this constantly changing situation with the virus evolves. I stand by much of what has already been stated; use common sense, keep distance from people, don’t touch your face, and wash your hands a lot with soap and water. And if you do become ill, isolate yourself until you understand which virus is afflicting you.

I have never been in politics – never run a town, city, or state. I don’t know what they spend their time doing, but I am pretty sure it is (or should be) spent understanding the current condition of the infrastructure and how to repair or upgrade it, along with understanding the needs of the citizens. So I never ran a town, but I was responsible for an industrial complex of 5,000 people. Responsible for the electrical distribution system, heat, air conditioning, energy use, overhead budget, gasses and compressed air as well as the buildings themselves. Underground, overhead, inside, or outside it was all my responsibility to ensure the people working their had a safe and functional place to work, that the place fully supported the work, and that the investment of the company into our facility was fiercely protected by my team.

In every situation at work (and there were plenty of them!) I have always been one to not panic; gather the facts, understand the scope, assess the options, and decide on a direction of action. But I was also one to spend time, a lot of time, an inordinate amount of time some might say, assessing risk. Where could the next catastrophic failure occur? And what could my team do to prevent it? And I know for a fact that we were extremely successful at recognizing, identifying, understanding, and mitigating potential or even probable, future catastrophes. Repairing or replacing whatever it might have been before it failed, installing sensors to monitor system conditions to alarm upon a deviation from nominal, or installing backup components.

But there were two kinds of events that while you could prepare for, you could not predict: hurricanes and blizzards. I spent hundreds of hours developing a detailed storm plan, one that covered both summer and winter. It began with major tasks and drilled down into the seemingly trivial; it divided the entire facility into manageable zones and then provided a specific name for each zone – a primary and a backup – with work, cell, and home phone numbers. It was phased by current and forecasted wind speeds. It detailed who needed to leave the facility, and when, as well as who needed to stay. It was a work that spanned years of constant effort.

And that storm plan got a lot of use – New England gets more than its fair share of storms year ‘round. And invariably, as the forecast grew grim and it was clear that it was an inescapable fact that we were going to get hit, the management team would convene in the main conference room and look at me and ask, “so what do we need to do?” Every time. Most never read the plan; those who did more or less skimmed it and didn’t really get much out of it. Every time. But there is some logic behind that, interestingly enough. Every storm is different; winds come from varying directions, storms arrive with high tide or low tide, snow can be wet and heavy or light and fluffy, and a myriad of other variables including timing – day or night, weekday or weekend. So every storm was different and the plan did not, could not, get down to that level – ultimately you had to wait and see what unique combination of weather we would get.

But they all knew the basics of the drill; from repetition they knew all the steps they had to do regardless of the specifics of the storm: bring material indoors, inspect their buildings for open or broken windows or doors, secure equipment, etc. So the blank looks in the storm meeting were actually somewhat misleading – most of them had already completed most all of the required steps – plan or no plan, they knew what to do.

But the catastrophe was not always a storm. I recall one event in particular. I was at my desk and my cell phone rang – it was one of my supervisors from another area of the plant; he was breathless and panting heavily as the words erratically burst from his mouth on alternate gasps: “Explosion….fire….so much smoke…Bay 1….power center 1…can’t see…..” I had a remote control for one of the unmanned gates near my office; policy dictated that users of an unmanned gate had to wait for the gate to fully open before proceeding through it and then wait for it to fully close before driving away and leaving the gate. On this day, I was through the gate and half way down the road to the other building before the gate was even half open. I think I set a land speed record getting from one part of the facility to the other (over public roads). The whole time I was driving I could see thick black smoke pouring out of the building.

Seconds later when I arrived at the huge doors leading into the cavernous building, there were hundreds of people running out of the building. The fire alarm was going off, fire and rescue vehicles from our company were arriving (town fire and rescue were still ten minutes or so away), and my supervisor was frantically waving to me from the door. He gave me a ten second rundown: a 2500-amp high voltage transformer and switchgear – one of seven serving the building – had shorted out and was decidedly on fire. I asked him if the power was still on; the building is fed at 12kV in series between the seven load centers from a single HV switch outside. He indicated that it was still energized so my first task was to run to the switch, unlock it, and open it to kill all power to the building.

I then ran inside over to the fire to try to understand if it was salvageable, and within 10 seconds it was clear to me that it was a total loss. I met up with our fire chief, and the town fire commander, and informed them that the power was indeed of and they were safe. I also advised them that we should shut off the bulk flammable gas distribution to the building as well and received a request to go make that happen, which I did.

Within a short amount of time the fire was extinguished. Senior management had convened a makeshift war-room in an adjacent building and I was requested to report on current conditions; we had close to a thousand people standing around outside and needed to quickly decide what to do with them as well as with second shift. By this point, I had been able to take a fairly close look at the still smoldering remains and had a good idea of what was damaged and what was not. The transformer and high voltage switch had not been affected by the fire – only the 480V switchgear had been damaged/destroyed. And most importantly the HV cabling that ran from power center to power center was clearly intact. And that meant it was likely we could restore power to the building and still leave the destroyed power center isolated.

After a lot of explanation and discussion, I was asked to develop a plan to safely re-energize the building. I huddled up with my team and we put together the plan of who, when, what, and where, and once I made certain that everyone on my team was in agreement, went back and presented it to the VP who was overseeing the event. The plan was approved and we went off and executed it was drawn up. It included a lot of testing and validation throughout the system that it was safe to re-energize which took a fair amount of time to complete, but finally we were ready – I held my breath (just a little) and threw the switch. That provided 12kV power to each of the seven power centers but nothing else as the HV switch at each one of them was open. My electrician and I then went from center to center, closing the HV main which energized the transformer, and then the low voltage main to provide restoration of power. Slowly the lights came back up, motors began running, and the building came back to life, with the exception of half of the first bay that was formerly powered by the now destroyed and dead power center.

That action allowed second shift to report to work as scheduled. While the power center was still smoldering, I had called one of my engineers and requested him to beg, borrow, or steal a 1mW rental trailered generator. He came through like the exceptional professional he was and within three hours of the initial call reporting the explosion, a tractor trailer with a massive 1mW generator, complete with load circuit breakers and miles of cabling. And by 9PM that evening we had every distribution panel in the affected area wired back to the generator and energized. The building was 100% functional once again.

Later that week I was summoned to our Groton home plant to meet with the president and explain exactly what had failed as well as why it had failed, followed by my recommendation on what to do to ensure it would not happen again. I gave him my plan and he gave me his verbal assurance that once officially presented through the proper channels with the required approvals, he would endorse it (which he ultimately did at just over a million-dollar price tag and a nearly one year period of execution).

My point is, despite the hundreds of hours pondering, assessing, considering, planning, and preparing for – there was no way to see this coming nor anyway to prevent it, and certainly no way to develop a recovery plan. There was just no way to know the extent of a failure and thus, how to recover from it without a major impact on production. In this case, it was literally formulated in minutes, once we knew the scope and extent of the damage.

I have never found looking back and criticizing helpful during an extreme event; save that for the post-event hotwash. During the event I need facts, data, and the best possible information to use for making a decision on how to react and what to do, as I just laid out in the fire story. Not a single person asked if it could have been prevented while we were working through the emergency. It is a responsible question but asking in the midst of it would be an irresponsible time to do so. Of course as I mentioned, the question was asked by the president, but after the event. And if you are wondering dear reader, here is the cause in all its technological detail. A 240/120V panel fed from a 600A breaker in this  power center developed a fault – a short if you will – that was due to corrosion; corrosion commensurate with our proximity to the salt water, the time in service and exposure to that sea mist, and the years of accumulation of extremely fine metallic grit from the 24/7 operation of welding and grinding. That electrical fault traveled back to the power center where the breaker tripped, but upon doing so failed catastrophically due to an arc flash. The arc flash spilled over into the adjacent cabling which propagated the fault throughout the switchgear resulting in its total catastrophic failure. Age was the primary culprit, coupled with hard service. I had requested funding for replacement power centers in that building for the previous four or five years, but the project was never funded.

So what’s my point and how does this relate to the current viral outbreak? While it has not been handled well by our government – the director of the CDC, the Secretary of Health and Human Services, and the president – now is not the time to point fingers and criticize. But politics and partisanship is doing exactly that; there is an extraordinary amount of effort by many to try and fix blame and criticize. If the intent is purely academic, it is best done when the issue has passed and all possible data and factual information is available. Doing so in mid-event is counterproductive and a waste of valuable resources much needed elsewhere, resources forced to explain and defend their actions.

I firmly believe in the competence, skill, ability, and professionalism of the overwhelming majority of the people employed in this country; from dump truck drivers and backhoe operators, to plumbers and electricians, to accountants and bankers, and so on. And doubly so in the people who provide their life services through the use of science. The work of scientists is extremely disciplined and bound by the integrity of empirical study. I find it nearly impossible to believe or accept the concept that our entire US medical community has “done nothing to prepare” or has been blindsided by this pandemic which has been slowly rolling across the globe; they’ve all seen it coming and I just don’t envision an entire community of scientists and medical experts just ho-humming and ignoring it. I am sure they have been working furiously for weeks, months even, trying to develop our plan.

But just like my storm plan, the vagaries and idiosyncrasies of this new virus are unknowns and the plan will likely not be adequate. And just like all the folks where I used to work, all the team members should already know what to do, at least the basics; we’ll get the specifics as the details become known. And no one criticized the me or the plan at that point. But afterwards, once the full fury had been felt and the full extent of damage was known, we’d sit down and everyone would present their views on what went right and what went wrong and how we needed to fortify our plan and future response actions. Call me into a roomful of people in the midst of a hurricane, whose sole agenda is to criticize that I didn’t line up enough dump trucks; call me in during an electrical fire to complain that I didn’t replace the switchgear before it happened? That only serves to distract me from my primary focus of containment and recovery. It’s too late to complain about inadequate prevention now – it’s already here. Don’t take me away from accomplishing what needs to be done to answer question and address complaints. Yet this is exactly what I see going on with our government; accusation, criticism, and complaint. Not the time, not the place.

We, the combined countries of the world, will get through this. Of that I have no doubt. People will die, sadly. But of course, to date this year there have been an estimated 52,000 deaths from the Type A and B influenza that have gone – mostly – unnoticed. This virus will add to that total, but its deaths will carry greater weight, greater significance, due to the publicity and politics of it all. In my humble opinion, politics, and agenda for the same, have no place in the midst of a crisis; it is time to put away partisanship and come together in a concerted, joint effort to get out factual, real, solid information to people, and to stop inciting panic and chaos with political bias and hidden motives. There’ll be plenty of time for that once we get through it.

So dear reader, avoid crowds and wash your hands and I pray my next post finds you all still well and healthy. Remember: plan and prepare, but don’t panic!

Leave a Reply