Previous incidents

This was a planned period where we rebuilt our core router and converted our network over to the rebuilt core. All went as planned.

At 7pm on 1/4/2023, we noticed a widespread network outage effecting the entirety of our ASN. We began to investigate and found issues with how a partner provider issued a blackhole for us. We worked to negate the blackholing, which in turn resulted in stuck routes throughout the DFZ. After working through some of the issues with stuck blackholed routes, we began to notice some issues presenting with IXPs we have presence in, likely stemming to the blackholing. After collective hours of our t...