Identity Services increased Response Time

Incident window: February 22, 2024 10:56 UTC
Impacted Cloud services:

• Foundry Identity Service

  • Foundry Identity Services Response Time increased in US-East Virginia region.

Impact Level : high

Identity Services is being investigated.

[2024-02-22 11:15 UTC] Foundry Identity Services Response Time increased. This could have impacted customers who are using MT identity from Virginia region.

[2024-02-22 11:45 UTC] Foundry Identity Services Response Time increased. This could have impacted customers who are using MT identity from Virginia region.

[2024-02-22 12:30 UTC] We are actively working on multiple parallel paths to mitigate this issue. Currently, we’re focused on mitigating the issue resulting in propagation delays, which we believe will also lead to an improvement in the overall error rates. We are hopeful that this mitigation will significantly improve the issue, and allow us to shift our efforts toward full resolution, and then fully understanding root cause and preventing recurrence. . We will continue to provide additional updates as we have them.

[2024-02-22 13:00 UTC] The mitigation has been successfully applied to the Identity Services in the US East Virginia region, resulting in the recovery of all instances. While the system stabilizes, customers may notice a temporary slowdown in response times for the next few minutes until the surge load subsides.

[2024-02-22 13:37 UTC] Resolved. Response times has receded and systems are running healthy, Our team is focused in investigating deeper to come up with a root cause analysis of the issue

Root cause Analysis:

The response delays stemmed from a surge in identity requests overwhelming the Identity service, causing a bottleneck in the authentication process and disrupted the services. This performance degradation of identity services arose from a surge in identity requests, exacerbated by clogged/stale connections stemming from a degraded service endpoint. Measures were taken to stabilize the service and alleviate degradation, including throttling down impacted nodes.

Mitigation Actions:

• Alert received by Cloud-Ops teams triggers investigation.
• Reports of service delays and degraded performance emerge from multiple customers. Requests, response times, and error rates witness an exponential surge.
• Initial analysis indicates potential requests crowding the Identity server.
• DDoS attack ruled out as a bottleneck in authentication process identified due to delays in service responses and all the requests were from legitimate sources.
• Restarting VM instances proves ineffective.
• Draining stale connections, restoring all nodes, yielded significant improvements.
• Throttling down impacted nodes controlled the request crowding in Identity.

Action Items:

To address the root cause of the issue we are proactively optimizing our platform to prevent clogged/ stale connections from degrading the Identify Service endpoint, and that work is targeted for completion by end of June 2024

Preventative measures:

HCL recommends to customers that grow above 10M sessions per year that they move to a dedicated single tenant Identity Environment. But we recognize that it is difficult for customers to plan this type of migration. HCL will work toward converting this recommendation to a requirement at certain traffic levels and more clearly communicate the risk of using shared identity for higher traffic applications. A dedicated Identity Environment provides isolation which. 1) allows tuning for load specific to one customer, 2) eliminates high traffic or issues from one customer impacting other customers, 3) provides additional security by being in a segregated environment, and 4) allows for easier troubleshooting and isolation of any issue. For more information on this please reach out to your respective Volt MX CEE or HCL Sales representative.