Resolved
The issue was resolved by our rate limit fix.
Monitoring
Record usage caused us to hit S3 rate limits. We've adjusted our invocation pattern with distributed calls and additional fallbacks. Less than 0.1% of active jobs were impacted. The issue is now resolved and we're continuing to monitor
Investigating
We're seeing elevated rates of 500 errors and are currently investigating.