Back to overview
Degraded

Network Instability

Aug 13, 2024 at 3:25am UTC
Affected services
po0.cr01.mci.trtnw.net
po1.cr02.mci.trtnw.net
po1.cr03.mci.trtnw.net
VX01
VX02

Resolved
Aug 13, 2024 at 4:03pm UTC

FRR v8 has seemed to resolve our issues, we have seen no relating concerns overnight following restoring this version of the router.

Updated
Aug 13, 2024 at 6:46am UTC

We have been monitoring traffic and network consistency for almost 3 hours now, things appear to be back to normal.

Updated
Aug 13, 2024 at 5:03am UTC

We've been monitoring traffic for an hour and have seen no drops in connection, no router crashes, or weird routing changes. We will continue monitoring throughout the night, however I'm going to mark this as resolved for now. Once again, we apologize about this. We're thinking revering back a few versions for our router software will be the solution. I've dug into errors similar in nature and some of the specifics symptoms we're facing have appeared to happen to other people who are on or above >v8.2.

Please feel free to reach out if you have any questions or concerns. We will do our best to triage tickets.

Updated
Aug 13, 2024 at 4:16am UTC

We've rolled back our core router to FRR v8, and things appear stable for now. We will continue to monitor the situation closely.

Created
Aug 13, 2024 at 3:25am UTC

We are aware of the recent network instability and are actively addressing the issue. Currently, it's unknown if these symptoms relate to the IPv6 issues from yesterday, but we are thoroughly investigating as the situation develops. We sincerely apologize for the inconvenience this has caused and are committed to resolving the problem as quickly as possible.