1 Introduction

Clustering involves deployment of multiple LoadMaster instances (nodes) as a single management and control domain where each node is actively passing traffic. If any of these nodes fail - the remaining nodes continue to provide service to the remote client, providing resilience.

Clustering supports scaling - LoadMaster nodes can be added to or removed from a cluster so that capacity can be dynamically adjusted to match business requirements. Each additional LoadMaster node which is added to the cluster can provide a linear increase in throughput and performance. The maximum number of nodes that can be added to a cluster is defined by your license agreement.

An example use case is - if a website experiences higher than usual traffic during a holiday period and the amount of traffic cannot be predicted in advance - often the capacity is over-provisioned to handle these requests that come in spikes. One of the simplest solutions is to deploy LoadMasters in a cluster and then add new nodes as the capacity requirements (concurrent connections and performance requirements) increase.

1.1 Document Purpose

The purpose of this document is to provide information on the LoadMaster clustering feature and step-by-step instructions on how to set up and configure clustering.

1.2 Intended Audience

This document is intended to be read by anyone who is interested in finding out more about LoadMaster clustering.

2 LoadMaster Clustering

2.1 Prerequisites

There are some prerequisites to be aware of before clustering LoadMasters:

A minimum of three LoadMasters are needed to set up clustering. However, four LoadMasters are recommended.

All of the LoadMasters in the cluster must be physically identical in terms of the hardware used. Clustering also works with Virtual LoadMasters but the Virtual Machines must have the same resources assigned; this includes CPU, memory and Hard Disk Drive (HDD). Once an empty machine is brought into the cluster, all parameters from the other machines (network interfaces, and so on), are copied over onto the new machine. Local addresses for network interfaces must then be set.

Requirements for clustering in a virtual environment (VMware, Hyper-V, and so on) are the same as those for High Availability (HA). In these environments:

- Ensure that Media Access Control (MAC) spoofing is allowed and enabled

- Ensure that multicast traffic flow is working in both directions between the devices

A 2-arm configuration is required.

2.2 Restrictions

Most functionality that is available in a single LoadMaster is available when a LoadMaster environment is clustered. The only restrictions are listed below:

All Virtual Services must operate at Layer 7

Transparency cannot be enabled

Subnet originating addresses are always enabled

Non-local Real Servers are not supported

HA mode cannot be used

VXLAN and IPsec tunneling are not supported

A 2-arm configuration is required

It is only possible to have a Virtual Service on the primary interface. Virtual Services on a second interface do not work.

2.3 Clustering Overview

With LoadMaster clustering, the load-balancing capability can be extended as needed by adding additional LoadMasters for the same Virtual Service. Traffic which is intended for a failed node is either passed on to an existing node or load-balanced across the remaining nodes.

The LoadMasters all work in parallel. If a LoadMaster node fails, traffic is redistributed amongst the remaining nodes. Health checking is performed on each of the LoadMasters.

Most administrative changes should be made on the shared IP address interface. These changes are then propagated to the LoadMasters in less than a second. Local administration of the individual LoadMasters can also be performed by accessing the WUI of the units directly. The WUI of the local LoadMasters contains limited configuration options.

2.3.1 How it Works

When traffic is forwarded to the cluster, the admin node load balances among the child nodes. When the child node receives the packet, it forwards it directly back to the client. This is done to efficiently allow for resource pooling of all nodes in the cluster. For this reason, you cannot configure transparency.

2.4 Configuring Clustering

If clustering is enabled on the LoadMaster, the Cluster Control menu item is available under System Configuration in the main menu of the LoadMaster WUI. If clustering functionality is not enabled and you would like to use it, please contact your Kemp representative.

2.4.1 Create a New Cluster and Add Nodes to it

To set up a cluster and add LoadMaster nodes to it, follow the steps in the sections below, in order.

2.4.1.1 Create a New Cluster

To create a new cluster, follow the steps below on a LoadMaster WUI:

1. In the main menu of the LoadMaster WUI, go to System Configuration > HA and Clustering.

2. This screen describes both HA Mode and Clustering. Select Clustering and click Confirm.

3. Click Create New Cluster.

4. Enter the desired shared IP address for the cluster.

This is the address that is used to access the shared WUI where configuration changes should be made.

5. Click Create a New Cluster.

6. The LoadMaster will then reboot. A prompt will appear asking to reconnect to the shared IP address.

7. Click Continue.

When the LoadMaster has rebooted, you are brought to the shared IP address.

The cluster is now set up. The next step is to add a LoadMaster to the cluster.

2.4.1.2 Add a Node LoadMaster to the Cluster

To add a node LoadMaster to the cluster, the node LoadMaster needs to become available and then have its IP address added in the shared IP address WUI. To do this, follow the steps below:

1. In the node WUI, go to System Configuration > Cluster Control in the main menu.

2. Click Add to Cluster.

The LoadMaster is now available to be added to the cluster.

3. Open the WUI of the shared IP address.

4. In the main menu, go to System Configuration > Cluster Control.

5. Enter the IP address of the node LoadMaster in the Address text box.

6. Click Add New Node.

This must be done while the node LoadMaster is available to join the cluster.

7. The node LoadMaster needs to reboot to finish the conversion to cluster mode. Click Continue.

8. By default, the new node is disabled. Click Enable to enable the node in the cluster.

When a node is enabled, it will not be immediately brought into rotation. It will only come online after it has been up for 30 seconds. The page may need to be refreshed to display any state changes. If there are no Virtual Services in the node, the node is in a Disabled state.

9. Click OK.

The cluster is now configured and a node has been added. To add more nodes, repeat the steps in the Add a Node LoadMaster to the Cluster section.

When all nodes have been added to the cluster, other configuration changes can be made - such as creating and configuring Virtual Services. These changes are propagated from the shared IP address to the LoadMaster nodes.

2.4.1.3 Configure the Network Interface(s)

Some configuration changes are needed on the interfaces. Refer to the sections below for further details.

2.4.1.3.1 Set the Cluster Shared IP Address

The cluster shared IP address needs to be set on any interfaces that are used to handle traffic. To do this, follow the steps below:

1. In the main menu of the shared IP address LoadMaster WUI, go to System Configuration > Network Setup.

2. Click the relevant interface link.

3. Enter the Cluster Shared IP address and click Set Shared address.

2.4.1.3.2 Enable the Use for Cluster Checks Option

The Use for Cluster checks option must be enabled on at least one interface. To enable this option, follow the steps below:

1. In the main menu of the LoadMaster WUI, expand System Configuration.

2. Click the relevant interface.

3. Tick the Use for Cluster Checks check box.

2.4.1.3.3 Enable the Use for Default Gateway Option

The Use for Default Gateway option needs to be enabled on each of the LoadMasters.

To do this, follow the steps below:

4. In the main menu of a node LoadMaster, expand System Configuration.

5. Click the relevant interface.

6. Tick the Use for Default Gateway check box.

2.4.2 Remove a Node LoadMaster from the Cluster

It is possible to remove a node LoadMaster from the cluster which will mark the node as unavailable for new traffic (that is, a Draining state) to facilitate the removal from the cluster. To do this, follow the steps below in the shared IP address WUI:

1. In the main menu, go to System Configuration > Cluster Control.

2. Click Disable on the node to be removed.

Before a node can be deleted, it must first be disabled. A node cannot be deleted if it is handling traffic.

3. The node will change to a Draining state. Wait for the node to finish draining and change to a Disabled state.

4. Click Delete to remove the node from the cluster.

5. Click OK to confirm the removal.

6. Click OK to the confirmation message.

When a node is deleted it becomes a regular single LoadMaster instance. If the LoadMaster is later added back in to the cluster, any configuration changes that have been made in the shared IP address will propagate to the node LoadMaster.

2.4.3 Changing the Cluster Parameters

The cluster Virtual ID and node drain time can be updated, if needed. Refer to the sections below for further information.

2.4.3.1 Updating the Cluster Virtual ID

When using multiple clusters or LoadMaster HA systems on the same network, the virtual ID identifies each cluster so that there are no potential unwanted interactions. To change the cluster ID, follow the steps below in the shared IP address WUI:

1. In the main menu, go to System Configuration > Cluster Control.

2. Click the Show Options button.

3. Change the Cluster Virtual ID value.

Valid values range from 1 to 255.

4. Click Set Cluster Virtual ID.

2.4.3.2 Updating the Node Drain Time

When a node is disabled, the connections that are still being served by the node are allowed to continue for the amount of seconds specified in the Node Drain Time text box. No new connections are handled by the node during this time. The Node Drain Time is set to 10 seconds by default. To change the Node Drain Time, follow the steps below in the shared IP address WUI:

5. In the main menu, go to System Configuration > Cluster Control.

6. Click Show Options.

7. Change the value in the Node Drain Time text box.

Valid values range between 1 and 600 (seconds).

8. Click Set Node Drain Time.

2.4.4 Backing Up and Restoring the Configuration

Backups can be taken and restored as usual in the System Configuration > System Administration > Backup/Restore section of the LoadMaster WUI. Backups and restores should only be performed on the shared IP address because the changes made there are propagated to the other nodes. When restoring, the individual LoadMasters are rebooted, one at a time.

There are certain settings that need to be set in the Virtual Services for them to work with clustering:

Transparency must be disabled

Subnet Originating Requests must be enabled

All Virtual Services must operate at Layer 7.

High Availability (HA) mode cannot be used

As a result of these restrictions, there may be issues with restoring a backup taken from a non-cluster LoadMaster on to a cluster LoadMaster and vice versa, because the settings may be incompatible.

2.5 Cluster-Wide Software Updates

The firmware on all LoadMasters in a cluster can be updated using the shared IP address.

Software updates require the LoadMasters to be rebooted. Functionality is maintained but existing connections will be lost for a short period of time.

To perform a cluster-wide software update, follow the steps below in the WUI of the shared IP address:

1. Get a LoadMaster firmware patch file.

You can download firmware patches from the Kemp Downloads page.

2. In the main menu, go to System Configuration > Update Software.

3. Click Choose File.

4. Browse to and select the firmware update patch file.

5. Click Update Cluster.

6. Click OK to continue.

7. Wait for the software update to be transferred and validated.

8. Click OK to install the software update.

9. A message will appear saying that the new software has been installed and the Cluster Control screen is displayed.

10. Each of the non-admin nodes need to be rebooted in order to activate the software. Click Reboot to reboot a node.

11. Click OK to the pop-up message.

12. When all non-admin nodes have been rebooted, the admin node needs to be rebooted. Click Reboot in the admin node row to reboot the admin node.

2.6 Clustering WUI Options

2.6.1 General WUI Options

If clustering functionality is enabled on a non-node LoadMaster, a Cluster Control menu option should be available in the WUI within the System Configuration section.

Before setting up clustering, clicking the Cluster Control menu item will give the option to either create a new cluster or add this LoadMaster to a cluster.

                                        

When logging in to the cluster, use the shared IP address to view and set the full functionality of the cluster nodes. Logging in to the local IP address of any of the nodes displays different menu options (see menus above). Logging in to one of the nodes directly is usually reserved for maintenance.

2.6.2 Cluster Control WUI Options

Create New Cluster: If setting up a new cluster, click this button.

Add to Cluster: Add this LoadMaster to an already existing cluster.

When the Create New Cluster button is clicked, the screen above will appear which prompts to set the shared IP address of the cluster. The shared IP address is the address which is used to administer the cluster.

When the Create a Cluster button is clicked, the LoadMaster reboots. A message will appear asking to reconnect to the shared IP address that was just set.

After creating a cluster, the Cluster Control screen in the WUI of the shared IP address will allow the addition of LoadMaster nodes into the cluster.

A LoadMaster can only be added to a cluster when the cluster is available and waiting to join the cluster. Refer to the Add a Node LoadMaster to the Cluster section for further information and steps.

The Cluster Control screen, in the shared IP address WUI, displays details for each of the nodes in the cluster.

Show Options: Clicking the Show Options button will display the Cluster Parameters section which contains two additional fields which can be used to set the Cluster Virtual ID and Node Drain Time. For further information, refer to the Cluster Parameters section.

ID: The cluster ID.

Address: The IP address of the LoadMaster node. If a second IP address appears in brackets after the first one - the second IP address is the IP address of the interface port. The IP address and status text is coloured depending on the status:

Blue: The node is the master node.

Yellow: The node is disabled.

Green: The node is up.

Red: The node is down.

Status: The status of the node. The possible statuses are:

Admin: The node is the primary control node.

Up: The node is up.

Down: The node is down.

Drain stopping: The node has been disabled and the connections are being shut down in an orderly fashion. Drain stopping lasts for 10 seconds by default. This can be updated by changing the Node Drain Time value on the Cluster Control screen. For more information, refer to the Updating the Node Drain Time section.

Starting: The node is starting

Disabled: The node is disabled - connections will not be sent to that node. If there are no Virtual Services in the node, the node is in a Disabled state.

Operation: The different operations that can be performed in relation to the notes:

Add new node: Add a new node with the specified IP address to the cluster.

Disable: Disable the node. Nodes that are disabled will first go through drain stopping. During the drain stopping time, the connections are shut down in an orderly fashion. After the drain, the node is disabled and no traffic is directed to that node.

Enable: Enable the node. When a node comes up, it will not be immediately be brought into rotation. It will only come online after it has been up for 30 seconds.

Delete: Delete a node from the cluster. When a node is deleted it becomes a regular single LoadMaster instance. If the LoadMaster is later added back in to the cluster, any configuration changes that have been made in the shared IP address will propagate to the node LoadMaster.

2.6.2.1 Cluster Parameters

When the Show Options button is clicked, the Cluster Parameters section appears. This section contains two additional WUI options - Cluster Virtual ID and Node Drain Time.

Cluster Virtual ID

When using multiple clusters or LoadMaster HA systems on the same network, the virtual ID identifies each cluster so that there are no potential unwanted interactions. The cluster virtual ID is set to 1 by default, but it can be changed if required. Valid IDs range from 1 to 255. Changes made to an admin Loadmaster propagate across all nodes in the cluster.

Node Drain Time

When a node is disabled, the connections that are still being served by the node are allowed to continue for the amount of seconds specified in the Node Drain Time text box. No new connections are handled by the node during this time. The Node Drain Time is set to 10 seconds by default, but it can be changed if required. Valid values range from 1 to 600 (seconds).

During the drain time the status changes to Draining until the specified drain time elapses.

When the drain time has elapsed the status changes to disabled.

2.6.3 Cluster Interface Options

There are a couple of fields relating to clustering in the Network Interface management screen.

To get to this screen, go to System Configuration > Network Setup in the main menu of the LoadMaster WUI and click one of the interfaces. The cluster-related options are described below.

Cluster Shared IP address

Specify the shared IP address which can be used to access the cluster. This is also used as the default source address when using Server NAT.

Use for Cluster checks

Use this option to enable cluster health checking between the admin LoadMaster and the nodes. At least one interface must be enabled.

Use for Cluster Updates

Select this option to use this interface to synchronize the configuration with all of the other nodes in the cluster.

2.6.4 Cluster-Wide Software Update WUI Options

The firmware on all LoadMasters in a cluster can be updated using the shared IP address.

This can be done in System Configuration > System Administration > Update Software by using the Update Cluster option.

After the firmware update patch has been uploaded, a new Reboot button will appear on the Cluster Control screen. When you click Reboot, a pop-up message will appear saying that the node is rebooting.

For more information, including step-by-step instructions on how to perform a cluster-wide software update, refer to the Cluster-Wide Software Updates section.

2.7 Application Program Interface (API) Commands

Kemp supports APIs that can be used to allow remote applications to access the LoadMaster in a simple and consistent manner:

RESTful API

PowerShell API

Clustering can be configured using API commands. For details on each of the commands that can be used, refer to the relevant API Interface Description document on the Kemp documentation page: www.kemptechnologies.com/documentation.

2.8 Troubleshooting

Logs are recorded to track certain clustering events, such as when a node goes offline. Viewing the logs can help to troubleshoot problems which may occur. To view the logs relating to clustering, in the LoadMaster WUI go to System Configuration > Logging Options > System Log Files. The logs relating to clustering are available in the Warning Message File and the System Message File.

Some examples of log entries relating to clustering are listed and described below.

The following log example shows a node failing and another node becoming the main admin node.

lb100 ucarp[2994]: [WARNING]Switching to state: MASTERlb100 ucarp[21751]: [CRITICAL]Partner has failed, becomingMASTERucarp[2995]: [WARNING]Switching to state: MASTERucarp[2995]: [WARNING]Preferred master advertised:going back to BACKUP stateucarp[2995]: [CRITICAL] Partnerhas failed, becoming MASTERlb100 l4d: HA state changed:now master.

The following log example shows a node going offline. This could be because the node was rebooted.

lb100 l4d: Node 3 now deadlb100 l4d: Cluster running ononly 3 of 4 nodes

The following log example indicates that a node has gone offline.

lb100 l4d: Cluster running on 3

of 4 nodes (1 disabled)

The following log example indicates that a node is alive, for example a disabled node has been enabled.

lb100 l4d: Node 3 now alive

References

Unless otherwise specified, the following documents can be found at http://kemptechnologies.com/documentation.

PowerShell, Interface Description

High Availability (HA), Feature Description

Last Updated Date

This document was last updated on 26 May 2022.