Several Funtoo Containers Offline (UPDATE)

[drobbins]

UPDATE: containers are back online -- see below for details on what happened.

Everyone,

Our datacenter in Taos, New Mexico experienced a very severe "fiber cut" in close proximity to the datacenter which severed its fiber links. This major fiber cut also has disrupted multiple ISPs and mobile carriers such as Verizon in Taos, NM.

This has resulted in a number of community containers going offline. This has not impacted Funtoo core infrastructure. The fiber cut is very severe and we expect at least 24 hours of downtime.

More info here: https://www.taosnews.com/news/severed-fiber-optic-line-causes-cell-phone-internet-outages-in-taos/article_cc7f41e6-03f7-11ee-9df2-fb4957e977a7.html

UPDATE from our datacenter:

What happened? At approximately 3:25 pm MT (GMT-6) yesterday a road construction crew caused a large fiber cut which knocked much of Taos, New Mexico's telecommunication infrastructure offline. This outage impacted both of our upstream networks in addition to local cell phone and phone service, as well as many local ISPs, including our sister company, TaosNet.  Lumen fiber technicians worked through the night to repair the broken fiber and are still at work on it.

Why did this happen?  Know that we're as shocked and devastated as you are by this extended outage and again, apologize profusely.  We were led to believe - by our upstream network providers - that our upstream network redundancy was complete, when in fact it was clearly not.  We have had fiber cuts to one of these fiber connections or the other over the last few years, and each time our traffic failed-over perfectly.  However, what this fiber cut revealed is that there is approximately a one mile stretch - of many hundreds of miles of fiber that our networks traverse - where both networks are in the same conduit. This is where the cut was.

The good news is that after detailed analysis we believe a network routing change can get us back to fully geographically redundant fiber paths.  We'll be working on implementing this in the coming weeks.

How did Brownrice get back online? Since the fiber outage was still not fixed, early this morning the CEO of a local ISP, Kit Carson Internet, reached out to us offering network connectivity.  Our network engineers, our TaosNet wireless team, and the team at Kit Carson Internet then formulated and executed a plan to build a high-speed wireless link between our two facilities and route traffic through Kit Carson Internet to our data center.  This brought all of our networking services back online. The amount of skill and know-how to pull this together within a single day was truly staggering.  And we can't thank Kit Carson Internet enough.

What are you doing to ensure this never happens again?  Now we have an active and tested, fiber-cut-proof backup connection in place and working.  It's not yet as fast as we'd like, and is currently experiencing some packet loss, but our higher speed fiber connections should come back online later tonight.  And once we are back to full strength we'll get the backup connection speed and packet loss sorted out.  And know that we will leave this backup connection in place forever, so this never happens again.