Incident Summary

On May 2nd, 2018 beginning at 12:05 UTC, the PingID and PingOne Dock applications began getting errors connecting to a database hosted on AWS RDS. A manual failover of the database node resolved the issue.

PingOne and PingID database infrastructure runs on AWS RDS (which is an Amazon managed service) and is configured for automated failover in the event of a primary database outage. During this incident, the primary database became unhealthy, but the RDS service did not failover the database properly and required a manual step to failover and restore service.

Customer Impacts

On May 2nd, 2018 beginning at 12:05 UTC, customers experienced the inability to reach the PingOne Dock (desktop.pingone.com) or authenticate with PingID MFA (authenticator.pingone.com) hosted in our North American data centers. Services began recovering at 12:31 UTC at which point PingID authentication sessions were successful. Due to the backlog of requests to the PingOne Dock, additional servers were deployed to handle the increased load and full service was restored at 12:49 UTC.

Incident Timeline

May 2, 2018 (all times in UTC)

• 12:05 - Monitoring systems detect issues with PingID and PingOne Dock services. On call SRE notified.
• 12:20 - Investigation shows increased number of errors connecting to database.
  
• 12:27 - On call SRE initiates manual failover of suspect RDS instance.
• 12:31 - PingID IDP services fully restored.
  
• 12:42 - Additional Dock servers deployed to handle increased load.
• 12:49 - PingOne Dock services fully restored.

Affected Services

• PingID Authenticator (.com)
• PingOne Dock (.com)

Resolution

Service restoration occurred after a manual failover of an AWS RDS database instance.

Ping Action Items

• Complete PingID RDS to Cassandra Migration.
• Improve PingOne Dock to reduce reliance on the read/write database.
• Investigate improvements to RDS database failover (jointly with AWS Engineering teams).