Write-up published
Read our write-up of this incident at https://app.notion.com/p/reductoai/Google-Cloud-Outage-Degradation-3a64a78a1c8781ffa45ad0fc7239b57c?v=2404a78a1c8780f390b1000c806bfd9f&source=copy_link
Resolved
This incident is now resolved. The upstream Google Cloud Vision OCR degradation recovered at approximately 09:48 PT / 16:48 UTC. Our fallback OCR paths have fully disengaged, error rates have returned to baseline, and Parse/Extract requests are operating normally. We will continue to monitor. Thank you for your patience.
Monitoring
GCP has rectified the issue and our fallbacks have started to disengage. We're seeing systems start to be healthy again but are continuing to monitor.
Identified
Multilingual parsing is good now ✅
Some documents with rotations may have degraded accuracy.
EU endpoint has an elevated failure rate and we're deploying a fix there now.
We're continuing to be in touch with Google Cloud to restore traffic away from fallbacks ASAP.
Identified
We've implemented a rough fallback due to the Google Cloud side service disruption causing a subset of requests to fail.
We are no longer seeing failing requests. There may be some accuracy degradation for multi-lingual document processing and rotated documents until the root cause is resolved.
Identified
We are seeing elevated failures on a subset of parse and extraction requests which touch an upstream cloud depednency and are actively mitigating.