PingID Service Interruption
Incident Report for Ping Identity
Postmortem

Incident Summary

On January 22nd, 2019 beginning at 23:22 UTC, customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) due to an internal networking issue in our Ohio region. A manual failover of the affected load balancers resolved the issue.

Customer Impacts

On January 22nd, 2019 beginning at 23:22 UTC, some customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) hosted in our North American data centers. This issue affected < 0.5% of transactions. Services were fully recovered at 23:33 UTC.

Customers that had implemented MFA bypass should not have been affected by this incident.

Incident Timeline

January 22, 2019 (all times in UTC)

* 23:22 - Monitoring systems detect issues with PingID services. SRE and Development notified.

* 23:26 - Investigation shows problem with load balancer state. Incident Command process initiated.

* 23:33 - SRE manual fails traffic to standby load balancer. PingID IDP services fully restored.

Affected Services

* PingID Authenticator (.com)

* PingID Server (.com)

Resolution

Service restoration occurred after a manual load balancer failover.

Ping Action Items

Deploy fix to ensure load balancer properly fails traffic over to standy servers for this edge case (Completed - 1/23)

Posted 5 months ago. Jan 24, 2019 - 15:41 UTC

Resolved
This incident has been resolved.
Posted 5 months ago. Jan 22, 2019 - 23:37 UTC
Investigating
Monitoring systems have detected an issue with Ping Identity's PingID Service. The Site Reliability Engineering team has been notified and is currently working the issue. We will update this message when the incident has been identified. Automated monitoring systems will update affected components and will resolve operational status as systems recover.

For additional questions please contact Ping Identity Technical Support by opening a case through The Community/Support site (https://www.pingidentity.com/en/account/sign-on.html), or follow this incident on https://status.pingidentity.com for real-time service updates.
Posted 5 months ago. Jan 22, 2019 - 23:22 UTC
This incident affected: PingID Services (PingID Authenticator - North America (.com), PingID Server - North America (.com)).