Incident Summary

PingOne for SaaS Apps Customers received an exception in the PingOne Admin Portal when editing manual customer connections. PingOne for Enterprise customers received an exception in the PingOne Admin Portal when editing Identity Repository Settings.

Customer Impact

Due to a software bug in the Admin Portal, customers who were not able to update to the new PingOne Universal Certificate prior to the certificate expiration were not able to update to the new valid certificate. This certificate is used for several purposes within the system: signing authentication and single-logout requests, and also for encrypting SAML that is sent from third-party providers to PingOne.

Incident Timeline

June 28th:
* Notification sent via email to customers with subject: "Action Required: Replacing the PingOne Universal Certificate" with steps to update the certificate and to contact support with questions.

July 21st (all times UTC): * 01:00 - PingOne Universal Certificate expires. * 01:45 - Customer Support contacted regarding customers not being able to manage connections (SEV3). * 02:17 - Customer Support pages On Call SRE. * 02:53 - On Call SRE escalates to Director of IOps - Investigation finds exception "more than one PingOne expired certificate available". * 03:25 - Director of IOps escalates to Sr. Director, SaaS Product Development. * 04:12 - Partial outage for Admin Portal posted to status.pingidentity.com. * 05:01 - Development confirms existence of bug in code. * 05:35 - Support confirms an expired certificate existed in the configuration of several customers. * 06:03 - Development confirms that this will not be addressed until the morning as it will take several hours to fix and test.

July 22nd (all times UTC): * 12:11 - Support requested to increase severity to SEV1. * 15:40 - Morning update posted to status.pingidentity.com. Support continues to manage customer expectations on case by case basis. * 16:27 - SEV1 issue escalated through Sales to Development and SRE. * 17:01 - Development shifts from looking at a code fix to mitigating the SEV1 issue. * 19:21 - SRE begins execution of DAO migration scripts.
* 22:13 - Migration process completed successfully and affected services restarted. * 22:16 - Support and Development confirmed fix is working and customers can manually switch certificates. * 22:29 - Support confirms last "EXPIRED CERTIFICATE IN USE" error at 22:13 UTC. * 22:44 - Update to statuspage.io stating customer issues are resolved.

Affected Services

• PingOne Admin Portal
• Single Sign-On (affecting customers that did not update the certificate prior to the expiration)

Resolution

Customers that had an expired certificate from 2014 as part of their configuration were migrated to a 2017 certificate. The expired 2014 certificate was removed, and customers were able to successfully update their connections to the new valid certificate.

Ping Action Items

• Improve expiration and notification procedure to allow customers more time to update configurations when certificates expire.
• Improve communication between internal teams (Sales, Support, SRE, Development) to ensure the impact of the issue is known.
• Build audit logs to allow the Support team to generate reports on certificate usage. This will enable regular follow up with customers if they are still configured with an expired certificate.
• Enhance internal testing plans to include expiring multiple certificates in TEST environment.
• Add support for multiple global signing certificates [SSD-5497 and SSD-5498].