PingOne/PingID Service Interruption (DNS Outage)
Incident Report for Ping Identity
Postmortem

Incident Summary

Beginning at roughly 19:29 UTC on 1/31/19, DNS lookups for the pingidentity.com domain resulted in "Host not found" results. Service was restored with a configuration change on our primary DNS provider.

DNS for the pingidentity.com domain is served by two external DNS providers. In the event of a failure with our primary provider, DNS is configured to be serviced by our secondary provider. In this case instead of queries failing they were returned with no data, and the switchover to our secondary provider did not happen as expected.

Customer Impact

Customers would have been unable to resolve, and therefore connect to, several of our hosted services (see affected services below). As this issue was caused by incorrect DNS lookups, the actual impact to customers will depend on how clients cached the DNS records. In some cases, there was no impact. In other cases, service was interrupted.

Incident Timeline

Jan 31, 2019 (all times in UTC)

* 19:29 - Monitoring systems detect issues with pingidentity.com services. Incident command process initiated.

* 19:36 - Configuration change applied on primary DNS provider.

* 19:40 - Services report as fully restored.

Affected Services

* PingID Service (.com)

* Token Processor Services (All Regions)

* Provisioning Services (.com)

* ADConnect & Routing Services (.com)

* Pingidentity.com Website

Resolution

Service was restored with a configuration change on our primary DNS provider. We are actively engaged with the vendor to determine what caused the issue.

Ping Action Items

* Engage DNS providers on ensuring proper movement of service to secondary provider.

* Audit and test the resiliency of our DNS infrastructure

Posted 19 days ago. Feb 01, 2019 - 17:49 UTC

Resolved
This incident has been resolved.
Posted 20 days ago. Jan 31, 2019 - 19:57 UTC
Monitoring
Services are recovering and the team is closely monitoring all services. Another update will be posted once everything is confirmed resolved.
Posted 20 days ago. Jan 31, 2019 - 19:44 UTC
Investigating
Monitoring systems have detected an issue with multiple PingOne Services. The Site Reliability Engineering team has been notified and is currently working the issue. We will update this message when the incident has been identified. Automated monitoring systems will update affected components and will resolve operational status as systems recover.

For additional questions please contact Ping Identity Technical Support by opening a case through The Community/Support site (https://www.pingidentity.com/en/account/sign-on.html), or follow this incident on https://status.pingidentity.com for real-time service updates.
Posted 20 days ago. Jan 31, 2019 - 19:35 UTC
This incident affected: PingOne Services (North America Critical Path (.com), Europe Critical Path (.eu), Australia Critical Path (.com.au), Admin Portal Monitor, Directory API, Single Sign-on, Single Sign-On (PingOne SSO for SaaS Apps/APS), Administration Portal, OAuth Service, Administration API, AD Connect & Routing Service, PingOne dock - North America (.com), PingOne dock - Europe (.eu), PingOne dock - Australia (.com.au), Directory Login - North America (.com), Directory Login - Europe (.eu), Directory Login - Australia (.com.au), Directory API - North America (.com), Directory API - Europe (.eu), Directory API - Australia (.com.au), Office365 Service - North America (.com), Office365 Service - Europe (.eu), Office365 Service - Australia (.com.au), SCIM Provisioning - North America (.com), SCIM Provisioning - Europe (.eu), SCIM Provisioning - Australia (.com.au)), PingOne For Customers (MFA API - North America, User Login API - North America, User Login - North America, Admin Login - North America), Pingidentity.com Website (Pingidentity.com Critical Path, www.pingidentity.com, docs.pingidentity.com, documentation.pingidentity.com, licensing.pingidentity.com), and PingID Services (PingID App, PingID Authenticator - North America (.com), PingID Authenticator - Europe (.eu), PingID Authenticator - Australia (.com.au), PingID Server - North America (.com), PingID Server - Europe (.eu), PingID Server - Australia (.com.au), Twilio SMS, Twilio REST API, PingID SDK Server - North America (.com), PingID SDK Server - Australia (.com.au), PingID SDK Server - Europe (.eu), E-mail OTP).