Service is operating normally: [RESOLVED] Elevated Packet Loss - AWS Outage

Summary

Between 9:42 AM and 11:44 AM PDT, we experienced elevated network packet loss impacting AWS Direct Connect connectivity in the AP-SOUTH-1 Region. Our engineering team was automatically engaged at 9:46 AM to begin investigating. By 10:51 AM, we understood the root cause to be a configuration change made to a system responsible for assigning routes to devices. As we gained confidence in our mitigation steps, we parallelized our efforts to further reduce the packet loss. The issue is resolved and t

Impact

Between 9:42 AM and 11:44 AM PDT, we experienced elevated network packet loss impacting AWS Direct Connect connectivity in the AP-SOUTH-1 Region. Our engineering team was automatically engaged at 9:46 AM to begin investigating. By 10:51 AM, we understood the root cause to be a configuration change made to a system responsible for assigning routes to devices. As we gained confidence in our mitigation steps, we parallelized our efforts to further reduce the packet loss. The issue is resolved and t

Lessons Learned

⚠AWS has experienced 73 incidents in the past year. This frequency suggests systemic reliability challenges that may warrant additional monitoring.

📊Incidents related to network, configuration, routing have occurred 249 times across all providers in the past year. This is one of the most common failure categories in cloud infrastructure.

💡This incident is categorized as: Configuration Error, Network / Routing. Consider implementing preventive measures specific to this failure category.