Super Bowl XLIX outage
Below is WPSD Local 6’s news release regarding the Super Bowl XLIX outages. You can also get a copy of the news release by clicking here.
News release regarding Super Bowl 49 outage
First and foremost, WPSD Local 6 regrets the interruption of the Super Bowl and we apologize to our viewers for the trouble. Below you will find a timeline of the night’s events and our findings from our continued investigation today.
On Sunday evening, February 1, 2015 during the third quarter of the Super Bowl the transmitter WPSD Local 6 faulted into a safe mode. This caused an interruption of the game for all viewers, except those who are Comcast customers in Paducah, KY. WPSD feeds Comcast via a fiber connection that is not impacted by outages at the transmitter site in Ballard County, KY.
Two engineers responded to the interruption of the broadcast. Our chief engineer made the half hour drive to the transmitter site, the second began the remote restart process for the transmitter. The high voltage transmitter resumed after going through its normal start-up cycle. This is a 12-15 minute cycle.
The game broadcast continued only to disappear minutes later when the transmitter again faulted into a safe mode. Both of these outages happened before the chief engineer arrived at the transmitter site. Again, the second engineer began the restart process, and again the high voltage transmitter went through its normal 12-15 minute cycle to resume operations.
Once inside the transmitter building the chief engineer found the transmitter operating normally, with no fault indications. He spent several minutes observing the operation and going through transmitter set up protocols. It was during this period that a third fault happened, and he was able to see the high voltage fault. He also witnessed a three-phase power outage with another piece of equipment. This was his first clue that the failure was not in our equipment, but rather tied to the electrical service WPSD Local 6 receives from Jackson Purchase Energy Corporation. The chief engineer called JPEC to report the problem and was informed that another customer had already called in the problem. During this third stoppage the transmitter performed as designed, shutting down and beginning its startup cycle.
WPSD has a backup emergency generator at the transmitter site. The chief engineer started the generator and prepared to switch off the JPEC power grid. By this time WPSD had restored the broadcast. The chief engineer checked the game status and made a decision to not switch off the JPEC power grid as the game was in the final two minutes and if the transfer did not work the broadcast could be lost for another 15 minutes. The game concluded under JPEC power.
A dispatcher from Jackson Purchase called the chief engineer around 9:45 p.m. Sunday to explain the trouble the utility was having and how it was impacting customers, including WPSD Local 6. Knowing the problem was not fixed and that the game had concluded, the chief engineer threw the transfer switch taking WPSD off the JPEC power grid and putting us on emergency generator service.
Our follow up investigation Monday, February 02, 2015 has found the following:
1. Jackson Purchase Energy discovered a bad “stinger on a capacitor bank” in rural Ballard County.
This bad “stinger” affected power to 260 customers, WPSD Local 6 was one of those. JPEC described the stinger as a piece of wire that goes from the power line to the capacitor. The capacitor helps stabilize voltage along the circuit. JPEC tells us that the “stinger” was broken and flopping in the wind when the repair crew arrived. This caused shorting along the power circuit. For homeowners this was seen in flickering lights, for our high voltage transmitter it was severe enough to trip safety measures.
2. WPSD Local 6 never suffered a power outage from JPEC, rather we experienced what our chief engineer describes as “dirty power.” These were power dips and spikes that were so quick and short in duration that they did not trip our automatic transfer to generator power, but were severe enough for the high voltage transmitter to shut itself down in safe mode. This is the way the transmitter is designed and it worked accordingly thus safeguarding against more severe equipment damage.
3. WPSD believes the bad “stinger” on the JPEC grid caused these dips and spikes in our service.
4. It wasn’t until the third bout with “dirty power” that a three-phase outage was recorded by equipment at the transmitter site. This gave us the clue that power supplied by JPEC was the cause for our trouble.
Again, we offer our sincerest apologies for the interruption of the broadcast. It was not a problem we could have foreseen. And when it arose we responded without hesitation. However, there is nothing we could have done to prevent the problem as the disruption of the game was due to a failure within the Jackson Purchase Energy Corporation power grid.