IPv6 related issues appear to be resolved at this time.

We were alerted to a memory leak on our core router, likely relating to CheckMK. We will be restoring a previous backup of the router without CheckMK installed which should resolve the issues relating to IPv6 not routing externally.

Post-mortem Report:

Incident Summary:

We experienced a critical issue with one of the main routers in our Autonomous System (AS), which led to a significant network disruption. The problem was traced back to an unexpected kernel panic on the core router. This kernel panic was caused by a combination of factors, including prolonged high load, incomplete automatic updates that required manual intervention, and several other issues that accumulated over time.

Incident Response:

Initially, we attempted to reboot the router after noticing that IPv6 traffic was routing internally but not externally. While this action temporarily restored service, it did not fully resolve the issue. We then decided to reroute all IPv6 traffic through external sessions (SIX, MICE, GPC) and completely rebuilt the core router from scratch. This involved reconfiguring dependencies and restoring the router’s configurations.

Outcome:

Once the rebuild and reconfiguration were completed, we brought the network back online. However, there was a brief latency spike due to the high volume of traffic being processed at once. As of now, network traffic has returned to normal levels, filters are rebuilding as expected, and both internal and external IPv6 routing has been fully restored.

If you continue to experience any issues, please don't hesitate to contact our support team. We are committed to ensuring the stability and reliability of our network.

We are still actively working through this issue.

We are pushing a fix to one of our core routers, we will keep you posted with the results.

We acknowledge that there are currently some issues affecting incoming IPv6 traffic. Please rest assured that our team is actively investigating the root causes and is diligently working to resolve these concerns as swiftly and effectively as possible. We understand the importance of reliable network performance and are committed to addressing this matter with the utmost urgency. Thank you for your patience and understanding as we work towards a solution.