PingOne Service Degradation - High Error Rates
Incident Report for Ping Identity
Postmortem

Incident Summary

To resolve an edge case discovered after enabling multi-region routing in North America, new cookies were introduced in the PingOne Dock to refetch session users and groups. This code change caused cookie storage corruption for some end users. As a result, the browser for these end users did not work properly and their Dock sessions were disrupted.

Customer Impacts

Users with cookie corruptions were unable to authenticate to the PingOne Dock.

Incident Timeline

May 7, 2019 (All times UTC)

* 08:29 Deployment of PingOne Dock.

* 08:45 Support receives reports of users being unable to access desktop.pingone.com successfully.

* 09:10 Issue confirmed and identified as due to recent software release.

* 09:19 Rollback completed to previous release. Service begins to recover.

* 10:04 Support completed confirmation from customers that service is restored after rollback.

Affected Services

* PingOne Dock (All Regions)

Resolution

Rollback to previous software version for PingOne Dock.

Ping Action Items

* Fix the cookie corruption issue and add a test case to catch it in the future.

Posted 2 months ago. May 14, 2019 - 20:52 UTC

Resolved
This incident has been resolved.
Posted 2 months ago. May 07, 2019 - 22:05 UTC
Monitoring
A fix has been deployed and the team is monitoring the system.
Posted 2 months ago. May 07, 2019 - 22:00 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted 2 months ago. May 07, 2019 - 21:51 UTC
Investigating
Monitoring systems have detected an issue with Ping Identity's Dock Service. The Site Reliability Engineering team has been notified and is currently working the issue. We will update this message when the incident has been identified. Automated monitoring systems will update affected components and will resolve operational status as systems recover.

For additional questions please contact Ping Identity Technical Support by opening a case through The Community/Support site (https://www.pingidentity.com/en/account/sign-on.html), or follow this incident on https://status.pingidentity.com for real-time service updates.
Posted 2 months ago. May 07, 2019 - 21:15 UTC
This incident affected: PingOne Services (PingOne dock - North America (.com), PingOne dock - Europe (.eu), PingOne dock - Australia (.com.au)).