System Status

System Status

Welcome to Ping Identity's system status site.

System Uptime

System Uptime

System uptime in the past 90 days.

Past Incidents Past Incidents

Welcome to Ping Identity's system status site.

Service Interruption (North America)
Incident Report for Ping Identity
Postmortem

Incident Summary

A change in a customer environment introduced sustained malformed requests to the PingID service in the early hours in the U.S. on Nov. 7, 2017. System availability was not impacted until approximately 13:38 UTC when traffic increased with the start of the work day on the US East Coast. Alerts were triggered at that time and some users experienced delays and retries in authentication. Shortly after the system became unresponsive for the majority of users. A fix for the malformed requests was deployed at approximately 14:15 UTC but did not fully resolve the issue. Traffic was then blocked from point of origin approximately 14:55 UTC. Service was fully restored for all users by 15:10 UTC. Investigation shows this was caused by a customer mis-configuration exposing an undiscovered defect in PingID system.

Customer Impacts

MFA with PingID for customers serviced by our North American data center was slow or unavailable.

Incident Timeline

Nov 07, 2017 (all time in UTC)

  • 07:00 - The PingID Development team begins to investigate increased error rates. Monitoring systems and automated tests show service still operating normally.
  • 13:38 - Monitoring systems report an increase in error rates and degraded customer experience. Some users experience delays and retries in authentication. Development team is engaged in triage.
  • 13:57 - Ping employees report failures in authentication.
  • 14:15 - Development deploys a software fix to mitigate the issue.
  • 14:55 - Customer IP blocked at the network level.
  • 15:10 - Services returned to normal.

Affected Services

PingID Service (North America)

Resolution

Blocking the source IP address of the problematic requests at the network level removed the condition that triggered the issue and eventually restored the service to normal. The malformed request activity was determined not to be malicious.

Ping Action Items

  • Deploy a fix for the input validation issue.
  • Enhance automated tests to include additional edge cases.
  • Improve ability to isolate customer traffic.
Posted Nov 09, 2017 - 22:08 UTC

Resolved
This incident has been resolved.
Posted Nov 07, 2017 - 15:30 UTC
Monitoring
We have identified the issue and applied a fix. The error rate is going down and the service is recovering. We will continue to monitor the service status.
Posted Nov 07, 2017 - 15:10 UTC
Update
The Site Reliability and Engineering team are still investigating this issue. We will provide additional update in 30 minutes.
Posted Nov 07, 2017 - 14:58 UTC
Investigating
Monitoring systems have detected an issue with North American PingID production systems. The Site Reliability Engineering team has been notified and is currently working the issue. Site Reliability will update this message when the incident has been identified.

For additional questions please contact support@pingidentity.com, or follow this incident on https://status.pingidentity.com for real-time service updates.
Posted Nov 07, 2017 - 14:24 UTC
This incident affected: PingID - United States (.com services) (PingID Authenticator, PingID Server).