Rolling deployment complete, issue was mitigated.

We are taking steps to ensure we handle rolling deployments in our end-to-end testing going forward to prevent this from occuring again.

Pipeline execution restored

Maintenance has completed

Fully resolved. No more latency impact in production. Post-mortem to come.

We immediately rolled back the problematic production deployment

Fully resolved.

New deployments rolled out and latency was resolved.