Incident Summary

Google Firebase Cloud Messaging experienced an outage, which caused push notifications in PingID Mobile Authentication App to stop working on Android devices.

Customer Impact

Some customers using Android devices could not receive push notifications for PingID Mobile Authentication App.

The root cause from our provider can be found at this URL: https://status.firebase.google.com/incident/Cloud%20Messaging/17005.

Incident Timeline

• 1425 - Google reports issues with Firebase Cloud Messaging service
• 1435 - Engineering and SRE teams confirm PingID service degradation
• 1524 - Google reports that they are still investigating the issue
• 1710 - Google reports that services should be restored
• 2121 - Google Firebase Status Dashboard updated showing services green

Affected Services

• PingID Android Mobile Authenticator
• In total, 11% of the push notifications on the PingID Android authenticator failed during the outage, where 89% succeeded.

Resolution

Resolution centered on Google restoring availability to Firebase Cloud Messaging service.

Ping Action Items

• Improve failover mechanism to detect all Push Notification errors and immediately fall back to One Time Password authentication.
• Ensure that when there is a manual workaround customers can initiate, it will be published on the status page (in this case, turn Swipe off on the mobile app).