Status | HCL Volt MX

HCL Volt MX Cloud status
Current status and incident report

Identity Services Outage affecting customers

Incident window: June 23, 2023 13:16 UTC to 14:00 UTC

Impacted Cloud services:

Foundry Identity Service
- Foundry Identity Services were running unhealthy in US-EAST Virgina region.

Impact Level : high

Identity Services is being investigated.

[2023-06-23 13:16 UTC] Foundry Identity Services were running unhealthy. This could have impacted multiple customers.

[2023-06-23 14:00 UTC] The services are restored back and it is running healthy.

[2023-06-23 16:21 UTC] We have captured all the logs and required information, our internal teams are investigating the issue.

[2023-06-26 17:21 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-06-27 18:45 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-06-28 14:00 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-06-29 12:45 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-06-30 10:00 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-07-01 15:01 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-07-02 16:01 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2023-07-03 10:01 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

Root Cause Analysis: During this incident It was observed there were evidences of failed communication between the Authentication VM hosted on US EAST region and one of the internal data source with IOExceptions followed by connection reset impacting the customers traffic with an authentication failure. IOExceptions were sourced from an unhealthy state of the connection objects in the VM.

Remediation: As this is a very very rare scenario to track down we have positioned additional automated observability with different anomaly conditions and are being observed spontaneously. In order to prevent such cases in future VoltMX Engineering is putting additional capabilities within is the product an improved visibility into the object’s health for better management of these objects in the VM.

Foundry V9 SP5 FP3 HF4 Release

Maintenance window: Monday, June 19, 2023 00:01 to 04:00

The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.

Impacted Cloud services:

Foundry Integration Server
- Support for bundling JAR into Custom Adapter added
- Specific type-casting issue in Offline Objects fixed

Impact Level : minor

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Cloud SSL Certificate Updates

Maintenance window: June 18, 2023 00:01 to 04:00 UTC

Impacted Cloud services:

Cloud SSL certificates will be updated for the following services
- App services (*.hclvoltmx.net)
- Dedicatated Identity services (*.auth.hclvoltmx.net)
⚠️ If your application uses public key pinning application updates should be made but are not necessary at this time. Customers that are using public key pinning should include the additional certificate pins in the JSON below.
⚠️ For customers using full SSL certificate pinning, please open a support ticket requesting a copy of the updated HCL certificates.
⚠️ If you are using full SSL certificate pinning, your applications must be published before the this HCL SSL certificate update on the cloud servers or the applications may no longer be able to connect to HCL servers.
If necessary, you can submit your applications for expedited approval (e.g., Apple has an expedited approval process for critical bugs, or in this case, pinned certificates).

Impact Level : high

Customer applications that have pinned the full SSL certificate must be updated as described above prior to this maintenance window. Customer applications that have not pinned SSL certificates will not be affected and will experience no service disruptions during this maintenance window.

Customers using public key pinning should include the updates to the pins listed below in the next publish of your mobile applications to the relevant app store. This set of pins also include the pins for additional backup SSL keys. HCL will be rotating certificates again later this year.

When mobile applications add the additional security of pinning a certificate for VoltMX servers, the application is built with information that allows it to use only locked down certificates. Verification of the SSL certificate, in combination with building the mobile application in protected mode, provides the best security for your mobile application.

The json below contains the necessary pins for Public Key Pinning of the HCL VoltMX Cloud production SSL certificates.

{
    "*.hclvoltmx.net"          : [
        "yjg/9eaEB+Lh68+DeUvX0cOb/+OMDqs3d15snWwRzYs=",
        "nV05xy/8ejyoGXvFV/tL+8JavIZ6O7rfU5Z/3RO2N4s=",
        "RVIUssObuqwIn/jA6zSNWqdU++8wxHfpm2uJrIJWrOM=",
        "q/xZuNZkwiNZ72g+gp7WuEvWMed1T0qTUcr47YpoMWM=",
        "NKvrJodATL1podc1DZ0Drd9lzf7SWEmK8oPL7l2g76g="
    ],
    "*.auth.hclvoltmx.net"     : [
        "CAMXj/DsquaCqbXLe7vpoH18P5ThihnLTVi0wJAr7FE=",
        "yjg/9eaEB+Lh68+DeUvX0cOb/+OMDqs3d15snWwRzYs=",
        "nV05xy/8ejyoGXvFV/tL+8JavIZ6O7rfU5Z/3RO2N4s=",
        "RVIUssObuqwIn/jA6zSNWqdU++8wxHfpm2uJrIJWrOM=",
        "q/xZuNZkwiNZ72g+gp7WuEvWMed1T0qTUcr47YpoMWM=",
        "NKvrJodATL1podc1DZ0Drd9lzf7SWEmK8oPL7l2g76g="
    ],
    "*.messaging.hclvoltmx.net": [
        "RD7OME7ESYWLNadwtTvCqO+DruprSlRPgi0nmrsXPjI=",
        "yjg/9eaEB+Lh68+DeUvX0cOb/+OMDqs3d15snWwRzYs=",
        "nV05xy/8ejyoGXvFV/tL+8JavIZ6O7rfU5Z/3RO2N4s=",
        "RVIUssObuqwIn/jA6zSNWqdU++8wxHfpm2uJrIJWrOM=",
        "q/xZuNZkwiNZ72g+gp7WuEvWMed1T0qTUcr47YpoMWM=",
        "NKvrJodATL1podc1DZ0Drd9lzf7SWEmK8oPL7l2g76g="
    ],
    "Amazon"           : "JSMzqOOrtyOT1kmau6zKhgT676hGgczD5VMdRMyJZFA=",
    "Amazon Root CA 1" : "++MBgDH5WGvL9Bcn5Be30cRcL0f5O+NyoXuWtQdX1aI=",
    "Go Daddy Secure Certificate Authority - G2": "8Rw90Ej3Ttt8RRkrg+WYDS9n7IS03bk5bjP/UXPtaY8=",
    "Go Daddy Root Certificate Authority - G2"  : "Ko8tivDrEjiY90yGasP6ZpBU4jwXvHqVvQI0GS3GNdA=",
    "Starfield Services Root Certificate Authority - G2" : "KwccWaCgrnaw6tsrrSO61FgLacNgG2MMLq8GE6+oP5I="
}

References

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP3 HF3 Release

Maintenance window: Monday, June 12, 2023 00:01 to 04:00

Impacted Cloud services:

Foundry Integration Server
- Issue with Object listing from Salesforce adapter fixed
Foundry Identity
- Healthcheck API has been enhanced

Impact Level : minor

Foundry V9 SP5 FP3 HF2 Release

Maintenance window: Monday, June 05, 2023 00:01 to 04:00

Impacted Cloud services:

Foundry Integration Server
- Fixed an issue with the RDBMS adapter casting long numeric strings to number for Stored Procedures

Impact Level : minor

Reports Maintenance

Maintenance window: Sunday, May 28, 2023 06:01 to 10:00

Impacted Cloud services:

Reporting Services
- Upgradation of Report Server.
Infrastructure updates
- Security Hardening.

Impact Level : major

Major downtime is possible for the impacted Cloud Reporting services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Foundry V9 SP5 FP3 HF1 Release

Maintenance window: Monday, May 22, 2023 00:01 to 04:00

Impacted Cloud services:

Foundry Integration Server
- Fixed persisting list of Server URLs when uploading JSON for Swagger
- Fixed an issue with expiry of Google Calendar backend tokens not being handled
Engagement
- Status Page performance issues resolved
AppFactory
- Android JAR signing issue (with JKS generated with JDK11 ) fixed

Impact Level : minor

Foundry V9 SP5 FP3 HF0 Release

Maintenance window: Monday, April 24, 2023 00:01 to 04:00

Impacted Cloud services:

Cloud Management Console
- Support for parameterized server URL for OpenAPI 3.0 has been added
- Fixed an issue with import of configurable Server properties under App Services
- Issue with uploading/viewing backend SSL certificates resolved
AppFactory
- Publishing multiple versions of Foundry applications is now supported
- Fixed an issue with build notifications listing incorrect triggered-by user
Foundry Integration Server
- Fixed an issue with Swagger adapter typecasting input params to String
- SAP Gateway adapter will be viewable for accounts that have pre-existing services of this legacy type in their Foundry instance already.
Foundry Identity
- SAP Gateway adapter will be viewable for accounts that have pre-existing services of this legacy type in their Foundry instance already.

Impact Level : minor

Foundry V9 SP5 FP2 HF3 Release

Maintenance window: Monday, April 10, 2023 00:01 to 04:00

Impacted Cloud services:

Engagement
- Hardened security against cross-site scripting vulnerabilities.

Impact Level : minor

HCL Volt MX Cloud status Current status and incident report

HCL Volt MX Cloud status
Current status and incident report