Foundry V9 SP5 FP15 HF13 Release

• Foundry Engagement Server

  • Added support for FCM HTTP V1 APIs in addition to legacy FCM

• Foundry Integration Server

  • Fixed issues with persistence of Swagger adapter configuration

• Foundry Object Services

  • Fixed an issue with floating-point numerals

• Foundry Identity Services

  • An issue with AD provider configuration has been resolved

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP15 HF12 Release

• Foundry Console (All services)

  • Added support for web-based apps to be created with services pointing to, and published to custom vanity-domains

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP15 HF9 Release

• Enterprise App Store

  • Resolved an issue with enabling authentication for applications deployed to EAS

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Identity Services increased Response Time

• Foundry Identity Service

  • Foundry Identity Services Response Time increased in US-East Virginia region.

Identity Services is being investigated.

[2024-02-22 11:15 UTC] Foundry Identity Services Response Time increased. This could have impacted customers who are using MT identity from Virginia region.

[2024-02-22 11:45 UTC] Foundry Identity Services Response Time increased. This could have impacted customers who are using MT identity from Virginia region.

[2024-02-22 12:30 UTC] We are actively working on multiple parallel paths to mitigate this issue. Currently, we’re focused on mitigating the issue resulting in propagation delays, which we believe will also lead to an improvement in the overall error rates. We are hopeful that this mitigation will significantly improve the issue, and allow us to shift our efforts toward full resolution, and then fully understanding root cause and preventing recurrence. . We will continue to provide additional updates as we have them.

[2024-02-22 13:00 UTC] The mitigation has been successfully applied to the Identity Services in the US East Virginia region, resulting in the recovery of all instances. While the system stabilizes, customers may notice a temporary slowdown in response times for the next few minutes until the surge load subsides.

[2024-02-22 13:37 UTC] Resolved. Response times has receded and systems are running healthy, Our team is focused in investigating deeper to come up with a root cause analysis of the issue

Root cause Analysis:

The response delays stemmed from a surge in identity requests overwhelming the Identity service, causing a bottleneck in the authentication process and disrupted the services. This performance degradation of identity services arose from a surge in identity requests, exacerbated by clogged/stale connections stemming from a degraded service endpoint. Measures were taken to stabilize the service and alleviate degradation, including throttling down impacted nodes.

Mitigation Actions:

• Alert received by Cloud-Ops teams triggers investigation.
• Reports of service delays and degraded performance emerge from multiple customers. Requests, response times, and error rates witness an exponential surge.
• Initial analysis indicates potential requests crowding the Identity server.
• DDoS attack ruled out as a bottleneck in authentication process identified due to delays in service responses and all the requests were from legitimate sources.
• Restarting VM instances proves ineffective.
• Draining stale connections, restoring all nodes, yielded significant improvements.
• Throttling down impacted nodes controlled the request crowding in Identity.

Action Items:

To address the root cause of the issue we are proactively optimizing our platform to prevent clogged/ stale connections from degrading the Identify Service endpoint, and that work is targeted for completion by end of June 2024

Preventative measures:

HCL recommends to customers that grow above 10M sessions per year that they move to a dedicated single tenant Identity Environment. But we recognize that it is difficult for customers to plan this type of migration. HCL will work toward converting this recommendation to a requirement at certain traffic levels and more clearly communicate the risk of using shared identity for higher traffic applications. A dedicated Identity Environment provides isolation which. 1) allows tuning for load specific to one customer, 2) eliminates high traffic or issues from one customer impacting other customers, 3) provides additional security by being in a segregated environment, and 4) allows for easier troubleshooting and isolation of any issue. For more information on this please reach out to your respective Volt MX CEE or HCL Sales representative.

Identity Services increased Response Time

• Foundry Identity Service

  • Foundry Identity Services Response Time increased in US-East Virginia region.

Identity Services is being investigated.

[2024-02-19 07:38 UTC] Foundry Identity Services Response Time increased. This could have impacted customers who are using MT identity from Virginia region.

[2024-02-19 8:33 UTC] The services Response time came back normal and it is running healthy.

[2024-02-19 08:34 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

Root Cause Analysis:

An automatic minor version update was applied to the identity database in the Virginia region. This update caused response delays at the database layer, leading to increased overall application response times.

Timelines:

[2024-02-19 07:38 UTC] Auto minor version upgrade triggered.

[2024-02-19 07:38 UTC] Excessive latency for Identity services in the Virigina region.

[2024-02-19 07:58 UTC] Auto Minor version upgrade completed.

[2024-02-19 08:10 UTC] Latency averages have improved to 20 seconds, but still impacting most APIs.

[2024-02-19 08:20 UTC] Latency averages have improved to 15 seconds, but still impacting most APIs.

[2024-02-19 08:33 UTC] Latency averages have returned to normal. Resolved and Identity services in the Virginia region are responding normally.

Foundry V9 SP5 FP15 HF8 Release

• Foundry Integration Server

  • Updates to payload encoding for special-characters

  • Updates to service management

• Cloud Management Console

  • Issue with applying date-range filters to Monitoring console addressed, and overall improvements to performance

• Foundry Object Services

  • OData4 support has been enhanced with additional verbs

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP15 HF7 Release

• Foundry Integration Server

  • Increased attribute size for operations to accommodate larger payloads

  • Added support for OAuth Identity Providers for Custom Data Adapters

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP15 HF6 Release

• Foundry Object Services

  • Fixes to Object service mapping

  • Updates to RDBMS connectivity check

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP15 HF5 Release

• Foundry Object Services

  • Enhancements to OData4 connectivity

• Database

  • JDBC driver package-name updates

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.