PingOne/PingID Service Interruption (DNS Outage)
Beginning at roughly 19:29 UTC on 1/31/19, DNS lookups for the pingidentity.com domain resulted in "Host not found" results. Service was restored with a configuration change on our primary DNS provider.
DNS for the pingidentity.com domain is served by two external DNS providers. In the event of a failure with our primary provider, DNS is configured to be serviced by our secondary provider. In this case instead of queries failing they were returned with no data, and the switchover to our secondary provider did not happen as expected.
Customers would have been unable to resolve, and therefore connect to, several of our hosted services (see affected services below). As this issue was caused by incorrect DNS lookups, the actual impact to customers will depend on how clients cached the DNS records. In some cases, there was no impact. In other cases, service was interrupted.
Jan 31, 2019 (all times in UTC)
* 19:29 - Monitoring systems detect issues with pingidentity.com services. Incident command process initiated.
* 19:36 - Configuration change applied on primary DNS provider.
* 19:40 - Services report as fully restored.
* PingID Service (.com)
* Token Processor Services (All Regions)
* Provisioning Services (.com)
* ADConnect & Routing Services (.com)
* Pingidentity.com Website
Service was restored with a configuration change on our primary DNS provider. We are actively engaged with the vendor to determine what caused the issue.
Ping Action Items
* Engage DNS providers on ensuring proper movement of service to secondary provider.
* Audit and test the resiliency of our DNS infrastructure
This incident has been resolved.
Services are recovering and the team is closely monitoring all services. Another update will be posted once everything is confirmed resolved.
Monitoring systems have detected an issue with multiple PingOne Services. The Site Reliability Engineering team has been notified and is currently working the issue. We will update this message when the incident has been identified. Automated monitoring systems will update affected components and will resolve operational status as systems recover.
For additional questions please contact Ping Identity Technical Support by opening a case through The Community/Support site (https://www.pingidentity.com/en/account/sign-on.html
), or follow this incident on https://status.pingidentity.com
for real-time service updates.
This incident affected: Pingidentity.com Website (Pingidentity.com Critical Path, www.pingidentity.com, docs.pingidentity.com, documentation.pingidentity.com, licensing.pingidentity.com), PingOne For Customers (MFA API - North America, User Login API - North America, User Login - North America, Admin Login - North America), PingID Services (PingID App, PingID Authenticator - North America (.com), PingID Authenticator - Europe (.eu), PingID Authenticator - Australia (.com.au), PingID Server - North America (.com), PingID Server - Europe (.eu), PingID Server - Australia (.com.au), Twilio SMS, Twilio REST API, PingID SDK Server - North America (.com), PingID SDK Server - Australia (.com.au), PingID SDK Server - Europe (.eu), E-mail OTP), and PingOne Services (North America Critical Path (.com), Europe Critical Path (.eu), Australia Critical Path (.com.au), Admin Portal Monitor, Directory API, Single Sign-on, Single Sign-On (PingOne SSO for SaaS Apps/APS), Administration Portal, OAuth Service, Administration API, AD Connect & Routing Service, PingOne dock - North America (.com), PingOne dock - Europe (.eu), PingOne dock - Australia (.com.au), Directory Login - North America (.com), Directory Login - Europe (.eu), Directory Login - Australia (.com.au), Directory API - North America (.com), Directory API - Europe (.eu), Directory API - Australia (.com.au), Office365 Service - North America (.com), Office365 Service - Europe (.eu), Office365 Service - Australia (.com.au), SCIM Provisioning - North America (.com), SCIM Provisioning - Europe (.eu), SCIM Provisioning - Australia (.com.au)).