MCNC Operations deliver during historic hurricane season

When all the hurricanes and tropical storms that formed in the Atlantic and eastern Pacific oceans this year are added together, the 2018 hurricane season was the most active season on record.

It was particularly unforgiving for many coastal communities – especially in the Carolinas.

Between Sept. 12 and Sept. 17, Hurricane Florence impacted much of North Carolina. As forecasted, the storm made landfall just south of Wilmington on Sept. 13. Even though at impact it had been reduced to a Category 1 storm, Florence hit the coast with high winds and record amounts of rain. Many communities received record-breaking rainfall, with more than 30 inches measured in some locations. At least 55 deaths were attributed to the storm and property damage and economic loss estimates have reached almost $17 billion.

The Federal Emergency Management Agency (FEMA) called upon MIT Lincoln Laboratory to use their state-of-the-art lidar (light detection and ranging) system to image the destruction in the region. MCNC provisioned a 10-gigabit IP connection from NCREN to Internet2 and made it possible for Lincoln Laboratory to process its lidar data without delay. Read Lidar scans over the Carolinas accelerate hurricane recovery.

MCNC operational teams and staff worked together leading up to Hurricane Florence by communicating with constituents on MCNC’s readiness and to gain an understanding of which locations planned to turn off equipment in advance of the approaching storm, according to MCNC Chief Operations Officer Tommy Jacobson. Additionally, he said, the fiber and facilities staff did a really good job getting out to all sites in the path of the storm for site inspections to make sure all facilities were prepped.

During the peak of the storm, Jacobson explained in a blogpost, there were 148 outages for endpoints served by NCREN. As with Hurricane Matthew two years prior, all MCNC hut locations east of I-95 at one point in time were operating via on-site emergency power generators due to facility power being unavailable. Generators ran for extended periods at MCNC’s hut in Wilmington, as well as in Fayetteville, New Bern, Morehead City, Pembroke, Whiteville, and Bolivia. NCREN core optical and IP networks did not experience an outage.

"Once again, another major hurricane impacting the state within the last 25 months and the core NCREN infrastructure worked just as designed,” added Jacobson. “All sites went to remote power as required and all core operations related to the core facilities, optical, and IP networks remained online and fully operational.”

Communication with MCNC senior management occurred on a daily basis, making sure everyone was aware of current status and requirements for assistance as warranted. The MCNC Business Continuity / Disaster Recovery (BCDR) Team also was activated and provided real-time communications to alert constituents to the current status of NCREN as the storm worked its way out of the state and operations returned to normal. 

Everette Teal, Director of Technology for Public Schools of Robeson County (PSRC), said he didn't think lightning could strike twice with two major storms in the Robeson County area in a two-year period – both of which destroyed central technology operations for the school district – forcing them to move their core data structure to a new, non-flood zone in order to get their system back online.

"MCNC came through immediately with a plan of action to move and restore Internet to our district in a very timely manner," added Teal. "Our service was back in operation in less than three business days with minimal loss of information and in full operation throughout the district. PSRC would like to give all the credit to the MCNC team for bringing some type of normalcy back to our staff and students."

Hurricane Florence remains a weather event that many in North Carolina will not forget or recover from for years to come. Moving forward, MCNC will continue to fine-tune its process and procedures during emergencies and weather-related events.

If you have some ideas, please reach out to us.