On May 25th, 2018 beginning at 09:27 UTC, customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) due to an issue with the deployment pipeline. Previous versions of the PingID application servers that were incorrectly carrying traffic were automatically removed from service. A restart of the load balancers forced traffic to the new servers.
An edge case was discovered where the deployment pipeline and the load balancers had different state information. The load balancers marked the current version as active, but directed traffic to a previous deployment. The deployment pipeline aged out the previous version (as designed), but in this case the previous deployment was incorrectly carrying live traffic. A restart of the load balancer resolved the issue and corrected the systems state.
On May 25th, 2018 beginning at 09:27 UTC, some customers experienced the inability to authenticate with PingID MFA (authenticator.pingone.com) hosted in our North American data centers. Services began recovering at 09:38 UTC with full restoration at 09:56 UTC.
Customers that had implemented MFA bypass should not have been affected by this incident.
May 25, 2018 (all times in UTC)