Degraded scale-out due to failed setups pulling from huggingface

lowReplicateJul 31, 2026 21:25

apicapacity

Capacity IssueAPI Issue

Summary

We're looking stable now, and with caching back on. Thanks again for your patience!

Impact

minor

Timeline

Jul 31, 2026 21:25

[identified] Models that pull from huggingface during setup have not been succeeding, which is resulting in scale-out delays and queue backups.

via statuspage

+2m

Jul 31, 2026 21:26

[identified] We have confirmed that turning off our caching layer for huggingface downloads allows models to successfully set up, so we are making that change for all models while we continue to troubleshoot.

via statuspage

+1h 1m

Jul 31, 2026 22:27

[monitoring] All affected hardware types have recovered and we are continuing to monitor and troubleshoot. Thanks for your patience!

via statuspage

+1h 12m

Jul 31, 2026 23:39

[resolved] We're looking stable now, and with caching back on. Thanks again for your patience!

via statuspage

Lessons Learned

⚠Replicate has experienced 39 incidents in the past year. This frequency suggests systemic reliability challenges that may warrant additional monitoring.

📊Incidents related to api, capacity have occurred 865 times across all providers in the past year. This is one of the most common failure categories in cloud infrastructure.

💡This incident is categorized as: Capacity Issue, API Issue. Consider implementing preventive measures specific to this failure category.