Resolved
We've verified this was fully mitigated. All systems are healthy.
Identified
We verified that we are no longer running into issues with open DB connections.
We are deploying a fix for the open idle transactions in the database.
Identified
We've identified an issue with many idle transactions being created when short jobs are submitted to the database. This was exposed only when we hit sufficient scale. We scaled down the maximum multi-tenant workers while we patch the idle transaction creation. At current scale, we're not seeing any more issues with the database running out of available connections. We're continuing to monitor.
Investigating
We're investigating an increased rate of failures on our API after a large burst in job volume.