System Status

System Status

Welcome to Ping Identity's system status site.

System Uptime

System Uptime

System uptime in the past 90 days.

Past Incidents Past Incidents

Welcome to Ping Identity's system status site.

PingID Service Degradation
Incident Report for Ping Identity
Postmortem

Incident Summary

On December 4, 2016, there were unexpected errors encountered in production during a planned PingID database migration from MySQL to Cassandra. Approximately 0.5% of PingID users were temporarily affected. As a result of these database migration issues, less than 0.51% of PingID users experienced configuration data corruption.

Customer Impact

Single-Sign-On was unavailable for approximately 0.5% of PingID MFA users as a result of this migration issue. The affected users could not SSO to applications that use PingID as MFA was blocked for the affected users. These users could not be unpaired or removed from PingID by an administrator due to the configuration data corruption.

Incident Timeline (MST)

  • 11:04 - Development teams discover errors during database migration
  • 11:49 - Discovery is escalated to the Site Reliability Team for assistance with investigation
  • 13:27 - Technical Support escalates to Site Reliability Team reporting PingID users are unexpectedly unpaired/disabled
  • 15:27 - Site Reliability Team validated database restore process in our non-production environment
  • 15:56 - Production database restore complete by the Site Reliability Team
  • 17:15 - Status posted with a status of “Identified"
  • 19:23 - Development teams started deploying a fix to production
  • 19:37 - Code fix completely deployed
  • 20:13 - Status post updating incident status to “Monitoring"
  • 08:21 (12/5) - Incident updated to “Resolved”

Affected Services

  • PingID Services
  • PingID Server (North America)
  • PingID Server (Europe)
  • PingID Server (Australia)

Resolution

A code fix was deployed that allowed impacted users to re-pair their devices to restore PingID functionality.

Posted Dec 14, 2016 - 00:43 UTC
Resolved
NOTE: Affected users may be prompted to re-register their device when attempting to use PIngID. In some cases, users will receive a general “Error” in their browser and no prompt is received by the mobile device. Users in this state will need to un-pair their device through the gears icon in the PingID application. After doing so, retrying the SSO request will prompt the user to re-register their device.
Posted Dec 05, 2016 - 15:21 UTC
Monitoring
Engineers have deployed a fix and full service has been restored.
Posted Dec 05, 2016 - 03:13 UTC
Identified
We've identified an issue impacting a subset of PingID users. SSO to applications that use PingID as MFA is blocked for the affected users as they can’t be unpaired or removed from PingID. Currently there's no workaround but we're in the process of deploying a fix. ETA of the fix is 2-3 hours.

For additional questions please contact support@pingidentity.com, or follow this incident on https://status.pingidentity.com for real-time service updates.
Posted Dec 05, 2016 - 00:15 UTC
This incident affected: PingID - Australia (.com.au services) (PingID Server), PingID - Europe (.eu services) (PingID Server), and PingID - United States (.com services) (PingID Server).
← Current Status Powered by Atlassian Statuspage