Beginning at roughly 19:29 UTC on 1/31/19, DNS lookups for the pingidentity.com domain resulted in "Host not found" results. Service was restored with a configuration change on our primary DNS provider.
DNS for the pingidentity.com domain is served by two external DNS providers. In the event of a failure with our primary provider, DNS is configured to be serviced by our secondary provider. In this case instead of queries failing they were returned with no data, and the switchover to our secondary provider did not happen as expected.
Customers would have been unable to resolve, and therefore connect to, several of our hosted services (see affected services below). As this issue was caused by incorrect DNS lookups, the actual impact to customers will depend on how clients cached the DNS records. In some cases, there was no impact. In other cases, service was interrupted.
Jan 31, 2019 (all times in UTC)
* 19:29 - Monitoring systems detect issues with pingidentity.com services. Incident command process initiated.
* 19:36 - Configuration change applied on primary DNS provider.
* 19:40 - Services report as fully restored.
* PingID Service (.com)
* Token Processor Services (All Regions)
* Provisioning Services (.com)
* ADConnect & Routing Services (.com)
* Pingidentity.com Website
Service was restored with a configuration change on our primary DNS provider. We are actively engaged with the vendor to determine what caused the issue.
* Engage DNS providers on ensuring proper movement of service to secondary provider.
* Audit and test the resiliency of our DNS infrastructure