Before You Upgrade (READ ME FIRST)

Please pay special attention to the issues below before you begin an upgrade to this LMOS release.

Generation of 4096-bit DHE Key

During an upgrade to this version of LMOS from a version prior to 7.2.53.0, a new 4096-bit DHE key is generated. On smaller LoadMasters, this can lead to significant CPU and memory consumption that could impact regular virtual service traffic. So, Kemp strongly recommends that this update be performed in a maintenance interval. 

Best Practices Cipher Set

In LMOS 7.2.52.0, the BestPractices cipher set was updated. If you are upgrading from a version prior to 7.2.52.0, this change is effective immediately after upgrade to this release. This change was made to improve LoadMaster security and conform to the latest industry best practices.

It is recommended, however, that you migrate your services as soon as possible to use the new BestPractices cipher set. For more information on the cipher suites removed from the set, please see the LMOS 7.2.52.0 Release Notes.

Supported Models for Upgrade

This release of LMOS is supported on the Hardware and Virtual models shown in the first three columns of the table below. It is not supported and should not be installed on any model listed in the two columns at right. This update patch can be applied to any supported model regardless of licensing (e.g., SPLA, MELA) or platform (e.g., hardware, local cloud, public cloud).

If your model number is not listed above, please see the list of End of Life models.

Upgrade Path

You can upgrade to this release of LMOS from any previous 7.2.x release. For full upgrade path information, please see the article Kemp LoadMaster Firmware Upgrade Path.

Upgrade Patch XML File Verification Notes

By default, verification of the digital signature on upgrade images is required in LMOS 7.2.50.0 and above. See the Update Verification Options setting under System Administration > Miscellaneous Options > WUI Settings. If the unit you are upgrading is set to require validation, you'll need to supply the XML Verification File supplied with this release.

Note that:

• In previous releases, two verification files were provided: one for pre-7.2.51 systems and one for later systems. This restriction has been removed with the 7.2.53.0 release; if upgrading from firmware 7.2.51.0 / 7.2.48.3 and above you can use the XML file provided with this release. If upgrading from any other firmware version you must following the upgrade path detailed in Kemp LoadMaster Firmware Upgrade Path article.
• LoadMasters running an LMOS version prior to 7.2.49 do not provide the option of XML file verification in the UI or API. If you are upgrading from one of these releases to this release, you can verify the digital signatures offline using a manual process documented on the support website.

Downgrading to Earlier Versions

Downgrading a LoadMaster running LMOS 7.2.55.0 to LMOS 7.2.51.0 (or a later release) can be performed using any desired Update Verification Options setting.

Downgrading to LMOS 7.2.50.0 or a previous release can only be done when the Update Verification Options setting is set to Optional or Legacy. When performing the downgrade, do not specify an XML file. If you want to verify the digital signature on the image before downgrading, you can do so using a manual process documented on the support website.

Downgrading on AWS

LMOS 7.2.55.0 now uses AWS Nitro-based instance types (see below). LMOS running on a Nitro instance cannot be downgraded to a release prior to LMOS 7.2.55.0. This issue will be fixed in the next LMOS LTS firmware release.

New Features

The following new features have been added to LMOS since the previous release.

Support for Newer AWS Instance Types

We have upgraded our AWS offerings to support 'Nitro-based' instance types as shown in the table below. The recommended instance type or each offering is listed first in bold.

Note that a fresh install of LoadMaster on AWS using one of the Nitro-based instances above cannot be downgraded to an earlier release. This issue will be fixed in the next LMOS LTS firmware release due in October 2021.

WAF: Clearing the False Positive Analysis Counters and Events

A Reset FPA Counters button has been added to the Web Application Firewall > False Positive Analysis page (which also clears the events table). If desired, the Download button at the top right of the Latest Events table can be used to download the current list of events before clearing.

WAF: Configurable OWASP POST Body Size

In previous releases, the maximum Request Body Size Limit for a POST body was hard coded to 1048576 bytes (1 MB). This setting is now configurable in the Virtual Service (and SubVS) API and UI settings. The default remains 1048576 bytes, with a supported range of 1024 bytes to 10485760 bytes (10 MB). This setting is available in a WAF-enabled Virtual Service under WAF Advanced Options. The Inspect HTTP POST Request Bodies option must be enabled before this new control is visible in the UI.

WAF: Remote Logging TLS Version

In previous releases, the updated WAF remote logging facility (Web Application Firewall > Export Logs) was not negotiating TLS versions above TLS 1.0. In LMOS 7.2.55, WAF has been modified to use the Certificates & Security > Remote Access > Outbound Connection Cipher Set setting for handshake negotiation.

Network Telemetry VLAN Enhancement

In previous releases, Network Telemetry could not be enabled on a VLAN with an IP address if the underlying interface was not also assigned an IP address. In this release, Network Telemetry can be enabled on a VLAN regardless of whether the underlying interface has an IP address.

Network Telemetry is an add-on package. After you upgrade to LMOS 7.2.55.0, do one of the following to get the latest package:

• If you're installing Network Telemetry for the first time, navigate to Network Telemetry in the LoadMaster main menu and click Install to get the latest add-on package.
• If you installed Network Telemetry on an earlier release, then after upgrading to LMOS 7.2.55.0 you can get the latest version of the add-on as follows:
  1. Go to the Other Downloads page on the Kemp website.
  2. Click on the Network Telemetry Flowmon Add-On link.
  3. The download page lists the add-on packages for both the latest GA release and for the LTS (Long Term Support) release. Click on the link for the 7.2.55.0 add-on.
  4. Once the download is complete, unzip the archive. There will be two files: the add-on image and an XML file.
  5. Navigate to System Configuration > System Administration > Software Update in the LoadMaster UI. The bottom section of the screen should look like this:

1. 6. Click the Browse buttons to upload the software package and the XML verification file.
   7. Once the files are uploaded, click Install Addon Package.
   8. Once the package is installed, click OK on the confirmation message that appears. The Version Installed in the screen above should now be 7.2.55.0.nnnnn.RELEASE.

Note: In a small number of cases, LoadMaster needs to be rebooted to complete the add-on upgrade. If the Flowmon add-on package appears in red text in the screen above, a reboot is required. Navigate to System Configuration > System Administration > Reboot System and click Reboot. Otherwise, the package is ready to use after you install or update it.

LoadMaster Dashboard Installer for Collector (Version 2)

A new version of the LoadMaster dashboard creation script for Flowmon Collector (supplied originally with LMOS 7.2.53.0) is available from the Other Downloads section of the support website. Key improvements in this version:

• An additional LMOS API script is provided to automate creation of the configuration file required to run the dashboard script. Like the dashboard script itself, this is a bash shell script that can be run from the Collector's SSH login shell.
• The dashboards created reflect LoadMaster settings (such as Alternate IP Address, Transparency, and Subnet Originating Requests) that affect the IP addresses LoadMaster uses to communicate with Real Servers and clients. Prompts are issues for any information that cannot be clearly identified using the API.
• SubVS traffic is now visualized on separate per-SubVS dashboards.

For complete instructions on using the scripts, see the documentation included in the download archive.

GEO: Capacity, Performance, and UI Enhancements

GEO capacity and performance have been improved in this release:

• In previous releases, the number of Fully Qualified Domain Names (FQDNs) that can be defined is limited to 256 total FQDNs. With this release, significant improvements to processing and performance have resulted in the removal of this limitation. The practical limit to the number of FQDNs supported will be determined by available system resources -- including the amount of load balanced traffic being handled by LoadMaster. As a rule of thumb, an FQDN with 64 IP addresses consumes about 2MB of memory.
• The global limit of 1024 IP addresses and records has also been removed. [Note: the limit of 64 IP addresses per FQDN remains.]
• Modifications to the FQDN UI support the above limitation changes and the UI should be generally more responsive than in previous releases.

Change Notices

SSL Renegotiation Disabled By Default

Starting with LMOS 7.2.55, the System Configuration > Miscellaneous Options > L7 Configuration > SSL Renegotiation setting will be disabled by default, as a recommended security best practice. There are many published vulnerabilities with renegotiation and TLS 1.3 removes support for it completely. Note that this change applies to both new deployments and upgrades.

Ciphers Use for Re-encryption

In previous releases, the ciphers used for re-encryption connections to Real Servers was not configurable. All re-encryption connections now use the same set of ciphers used by other outbound connections, as specified by the Certificates & Security >Remote Access > Outbound Connection Cipher Set setting.

Increased Size Limitation for SSO Custom Form Images

The size limitation for images provided in custom image sets for Forms Based single sign on has been increased from 256 KB to 1 MB.

RPS Limiting UI Removed for Non-Offloaded HTTPS Port 443 VSs

The QoS/Limiting option for rate limiting by HTTP Requests per Second (RPS) will no longer appear in the UI for HTTPS Virtual Services on port 443 with SSL Acceleration disabled. SSL Acceleration must be enabled or this option will not appear -- the SSL connection must be terminated on LoadMaster for this option to work.

Security Updates

Update OpenSSL to Version 1.1.1k

The version of OpenSSL on LoadMaster has been updated from 1.1.1 (no letter) to 1.1.1k, to address various issues in the previously supported release. See the OpenSSL 1.1.1 Release Notes page for more information on the differences between 1.1.1k and previous releases.

Strict Transport Security Header Settings

HTTP Strict Transport Security (HSTS) allows a server (in this case LMOS) to set a header in client responses that instructs the client to force all subsequent connections to use HTTPS and to disregard any attempt to load any resource in that domain (and possibly its subdomains) over HTTP.

The Strict-Transport-Security header has various associated settings, none of which were exposed in the UI in previous releases. With this release, all settings are available through both the API and the UI. In the UI, they are exposed as follows:

• The default maximum age of all Strict-Transport-Security headers set by LoadMaster is 31536000 seconds (365 days/1 year). This global value can be modified on the System Configuration > Miscellaneous Options > L7 Configuration page by setting L7 Security Header Age to the desired number of seconds. Two years (63072000 seconds) is a commonly used value; the largest value that can be set is three years (94608000 seconds).
• The content of the Strict-Transport-Security header can be customized for each Virtual Service in the SSL Properties section of the VS configuration:
  • Don't add the Strict Transport Security Header: This is the default value.
  • Add the Strict Transport Security Header -- no subdomains: Adds the header only to client responses in the domain, not for any subdomains.
  • Add the Strict Transport Security Header -- include subdomains: Adds the header to client responses in the domain and all subdomains.
  • Add the Strict Transport Security Header -- no subdomains + preload: Adds the header only to client responses in the domain, not for any subdomains; allow the use of HSTS preloading, if supported by the client browser.
  • Add the Strict Transport Security Header -- include subdomains + preload: Adds the header to client responses in the domain and all subdomains; allow the use of HSTS preloading, if supported by the client browser.

See the following links for more information and guidelines on setting the HSTS header; also see this explanation of HSTS preloading.

Single Sign On: SameSite and Secure Options

Single Sign On data connections in previous release didn't include either a "SameSite" or "Secure" parameter in the Set-Cookie header. With this release, the "Secure" parameter is now always sent and, by default, the "SameSite" parameter is not added. These options can be set globally or per-Virtual Service:

• The global setting on the System Configuration > Miscellaneous Options > L7 Configuration page can be set to the following values:
• SameSite Option Not Added (the default value, compatible with previous releases)
• None
• Lax
• Strict
• The Virtual Service setting appears under ESP Options when ESP is enabled and Client Authentication Mode is set to Forms Based. The default value at this level is the System Default setting, which means it's the same as the global setting. The other values shown above can also be set at the VS level.

Console Support for WUI Cipher Reset

The system console has been enhanced to support resetting the cipher set used by the LoadMaster UI, for use cases where setting a cipher set improperly may cause the UI to be unreachable. To use this facility:

1. Log into the system console using the hardware or hypervisor console capability, or via SSH.
2. At the LoadMaster Configuration menu, select Local Administration > Web Address > Restore Admin WUI access to default mode.

This command does the following:

• Resets the Certificates & Security > Admin WUI Access > WUI Cipher Set parameter to the default WUI cipher set.
• Resets the Certificates & Security > Remote Access > Self-signed Certificate Handling parameter to the default (RSA self-signed certs).

Certificate Chain of Trust for UI Authentication

The ability to specify the intermediate and Certificate Authority (CA) certificates to be used to validate a client certificate presented for login to the UI has been added to the API and to the Certificates & Security > WUI Access Options UI page. Controls have been added to the top of the page under Admin WUI Options that list all the intermediate and CA certificates currently installed on LoadMaster and allow you to select the certificate(s) that will be used to validate client certificates presented for login. Any client certificates presented whose chain of trust cannot be validated using the selected CA and Intermediate certificates will be denied access. The default is to check against all existing certificates.

Console CLI Security Update

The system console has been updated to close vulnerabilities present in the CLI in previous releases that could allow an already authenticated user to obtain a privileged shell. The CVE identifier for this vulnerability is CVE-2021-41068.

WUI Template Security Update

Validation has been enhanced for the upload of a Virtual Service Template to the system, to close a security vulnerability wherein a carefully constructed file can be uploaded as a template and create unwanted files on the filesystem. The CVE identifier for this vulnerability is CVE-2021-41069.

Issues Resolved

New Known Issues

<FQDN>:794 Uncaught ReferenceError: disp_addrr_elements is not defined
    at <FQDN>:794
(anonymous) @ <FQDN>:794

Existing Known Issues