Network Instability
Resolved
Aug 13 at 12:03pm EDT
FRR v8 has seemed to resolve our issues, we have seen no relating concerns overnight following restoring this version of the router.
Affected services
core1.mci.bb.tritan.host
core3.mci.bb.tritan.host
VX01
VX02
core2.mci.bb.tritan.host
Updated
Aug 13 at 02:46am EDT
We have been monitoring traffic and network consistency for almost 3 hours now, things appear to be back to normal.
Affected services
core1.mci.bb.tritan.host
core3.mci.bb.tritan.host
VX01
VX02
core2.mci.bb.tritan.host
Updated
Aug 13 at 01:03am EDT
We've been monitoring traffic for an hour and have seen no drops in connection, no router crashes, or weird routing changes. We will continue monitoring throughout the night, however I'm going to mark this as resolved for now. Once again, we apologize about this. We're thinking revering back a few versions for our router software will be the solution. I've dug into errors similar in nature and some of the specifics symptoms we're facing have appeared to happen to other people who are on or above >v8.2.
Please feel free to reach out if you have any questions or concerns. We will do our best to triage tickets.
Affected services
core1.mci.bb.tritan.host
core3.mci.bb.tritan.host
VX01
VX02
core2.mci.bb.tritan.host
Updated
Aug 13 at 12:16am EDT
We've rolled back our core router to FRR v8, and things appear stable for now. We will continue to monitor the situation closely.
Affected services
core1.mci.bb.tritan.host
core3.mci.bb.tritan.host
VX01
VX02
core2.mci.bb.tritan.host
Created
Aug 12 at 11:25pm EDT
We are aware of the recent network instability and are actively addressing the issue. Currently, it's unknown if these symptoms relate to the IPv6 issues from yesterday, but we are thoroughly investigating as the situation develops. We sincerely apologize for the inconvenience this has caused and are committed to resolving the problem as quickly as possible.
Affected services
core1.mci.bb.tritan.host
core3.mci.bb.tritan.host
VX01
VX02
core2.mci.bb.tritan.host