NKC Datacenter Internet Outage

On August 6 we experienced a significant Internet outage at the datacenter where our VELOXcloud services are hosted.  Before I explain what happened, let me give a quick disclaimer.  I'm describing the supposed causes of a serious outage.  The information I present was gathered from various sources and our own investiagation.  It is preliminary in nature, and all of the facts are not yet known.

Around noon a contractor began boring through 11th Street between McGee and Oak Streets for a new manhole.  The area they were drilling should have been free of utility cables; however, it was not, and the fiber that carries our Internet access along with a number of other data services was cut.  An investigation is underway regarding the accuracy of the cable location markings.

The fiber was cut at 12:49 PM and our INmon monitoring system identified the outage within just a few seconds.  Our upstream Internet provider and the owner of the fiber were engaged within minutes.  They have both stayed engaged and have worked diligently to resolve the problem throughout the night.  The problem was quickly identified as a fiber cut, and crews were retained to begin the process of fixing the fiber. 

Crews have worked diligently through the night and are still working as I write this to get all of the fiber-optic cable back in working order.  Thankfully they were willing to place priority on the fiber that contains our circuits and our access to the Internet returned to working order right at midnight. 

Please accept our sincere apologies for this issue.  It has drawn attention to a weakness within our network that we were already aware of.  In fact, we've been working on a solution to add redundancy in this area, but we hadn't placed a high priority on that because our Internet access has been stable.  After this extensive outage, we will be looking to add Internet redundancy to our NKC services in the near future. 

If you have any questions or concerns, please contact us.