Service impact: Elevated Packet Loss - AWS Outage

Summary

We have identified the root cause to be related to a change made to a configuration system responsible for assigning routes to the devices. We have begun the work to reduce the packet loss that is impacting AWS Direct Connect in the AP-SOUTH-1 Region, and we expect recovery to take place gradually over the next 30 minutes. As we gain confidence in these efforts, we will seek to parallelize our efforts to speed recovery. We will provide another update by 12:00 PM PDT.

Impact

We have identified the root cause to be related to a change made to a configuration system responsible for assigning routes to the devices. We have begun the work to reduce the packet loss that is impacting AWS Direct Connect in the AP-SOUTH-1 Region, and we expect recovery to take place gradually over the next 30 minutes. As we gain confidence in these efforts, we will seek to parallelize our efforts to speed recovery. We will provide another update by 12:00 PM PDT.

Lessons Learned

⚠AWS has experienced 73 incidents in the past year. This frequency suggests systemic reliability challenges that may warrant additional monitoring.

📊Incidents related to network, configuration, routing have occurred 249 times across all providers in the past year. This is one of the most common failure categories in cloud infrastructure.

💡This incident is categorized as: Configuration Error, Network / Routing. Consider implementing preventive measures specific to this failure category.