We are seeing full recovery.

Resolved.

Changes were rolled back and error rate dropped to normal rates.

During this window, we are applying several upgrades to our infrastructure. There may be some jobs which fail during this window which should be retryable.

All Studio runs should now be successful.

All inference services restored. New jobs will continue to succeed.

backend incident no longer affecting product experience or /pipeline invocation

Maintenance has completed

all services are operational now

We've root caused and resolved database issues. Services are fully operational.