Incident Summary

PingID stopped functioning correctly which prevented users from being able perform second factor authentication. The root cause of the outage was a data replication failure in the session management system. An unusual circumstance occurred where a failed node was replaced then rebalancing of session data stalled the entire system. Mitigation actions were taken and the system functionality was restored (see below).

Customer Impact

Customers were not able to utilize PingID during the duration of the outage.

Incident Timeline - Apr 17, 2017 (MDT)

• 1745 - PingID errors reported
• 1750 - Operations Team begins investigation
• 1751 - System monitoring indicates spike in HttpServerError 500
• 1753 - Web server stack trace shows problem connecting to the session management system
• 1756 - Internal escalation process initiated
• 1806 - Synthetic testing validates problem
• 1815 - Restarting web services reduces error rate
• 1817 - Load balancing mechanism marks all nodes as down
• 1822 - Heartbeat for all nodes return to normal
• 1832 - Status monitoring page updated
• 1836 - Service is restored
• 1851 - Status monitoring page updated

Affected Services

• PingID Services
• PingID App
• PingID Authenticator
• PingID Server

Resolution

Restarting the web services allowed the stateless session management system to fully recover.

Ping Action Items

• Improve error monitoring of synthetic tests to detect this type of failure sooner.
• Improve status update process and method.
• Change the PingID session management system implementation to be more resilient.