On January 22nd, 2019 beginning at 23:22 UTC, customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) due to an internal networking issue in our Ohio region. A manual failover of the affected load balancers resolved the issue.
On January 22nd, 2019 beginning at 23:22 UTC, some customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) hosted in our North American data centers. This issue affected < 0.5% of transactions. Services were fully recovered at 23:33 UTC.
Customers that had implemented MFA bypass should not have been affected by this incident.
January 22, 2019 (all times in UTC)
* 23:22 - Monitoring systems detect issues with PingID services. SRE and Development notified.
* 23:26 - Investigation shows problem with load balancer state. Incident Command process initiated.
* 23:33 - SRE manual fails traffic to standby load balancer. PingID IDP services fully restored.
* PingID Authenticator (.com)
* PingID Server (.com)
Service restoration occurred after a manual load balancer failover.
Deploy fix to ensure load balancer properly fails traffic over to standy servers for this edge case (Completed - 1/23)