Google Play introduced a new mandatory version requirement
We released compliant version 1.64.xx in early Jan and have been observing normal operations for 3 weeks and several minor release iterations
Incident:
On Feb 1 we started observing early signs of deteriorating detection on Android for a very small subset of user and started investigation
On Feb 4 volume of impacted users started approaching ~10% with disproportionate bulk of the affected users being on Samsung devices running Android 12 and 13
On Feb 6 impacted users count escalated to ~15% and thorough our investigation efforts we have isolated the issue to be in the transport layer of the app
On Feb7 we started testing potential fix and created backend workaround to facilitate the drive capture
On Feb 8 we started seeing mitigation measures take effect and we rolled out an app fix, percentage of users started seeing recovery at that time
On Feb 9 we created additional app update and rolled it out to 100%, effectively resolving the issue
Learnings:
We will be prioritizing transport layer robustness as a part of our ongoing product modernization
We were under-monitored in some areas and will focus on additional telemetry that would have sped up the detection and remediation
We will allow for more time to soak test mandatory Play Store target SDK updates
Posted Feb 10, 2023 - 20:52 PST
Resolved
We've been observing normal operations for some time now on versions 1.64.6 and above.
Posted Feb 10, 2023 - 12:15 PST
Update
We're observing normal operation and processing rates on the latest Android update available in Play Store. Please update to version 1.64.6 or above if you're still experiencing issues.
Posted Feb 10, 2023 - 08:29 PST
Monitoring
We've started rollout of a new version which includes improvements around drive detection stability and additional telemetry.
Posted Feb 09, 2023 - 23:11 PST
Identified
We have started putting remedial measures in place earlier today and observe a significant improvement in drive detection and processing operations.
Posted Feb 09, 2023 - 11:38 PST
Update
We are continuing the investigation and are pursuing multiple paths in parallel. Our current belief is that the issue is related to the mandatory Google Play target version changes. We will provide further updates as they become available.
Posted Feb 08, 2023 - 07:26 PST
Update
We see the problem to be prevalent on Samsung devices running Android 12 and 13. We're continuing investigation and will provide more updates as the become available.