An instance is repeatedly attempting to perform ~2,500 simultaneous file uploads. This is exhausting Apache process slots on the Phacility web tier and causing load balancers to fail health checks, leading to sites being inaccessible.
This problem is well understood, but applying an effective rate limit is difficult.
We've taken these steps to mitigate the issue until it can be fixed properly:
- Added additional web tier capacity.
- Temporarily disabled the "file.allocate" API endpoint.
We are pursuing a more durable fix based on rate limiting for this class of requests.