LoadMaster Clustering

 

1Introduction

Clustering involves deployment of multiple LoadMaster instances (nodes) as a single management and control domain where each node is actively passing traffic. If any of these nodes fail - the remaining nodes continue to provide service to the remote client, providing resilience.

Clustering supports scaling - LoadMaster nodes can be added to or removed from a cluster so that capacity can be dynamically adjusted to match business requirements. Each additional LoadMaster node which is added to the cluster can provide a linear increase in throughput and performance. The maximum number of nodes that can be added to a cluster is defined by your license agreement.

An example use case is - if a website experiences higher than usual traffic during a holiday period and the amount of traffic cannot be predicted in advance - often the capacity is over-provisioned to handle these requests that come in spikes. One of the simplest solutions is to deploy LoadMasters in a cluster and then add new nodes as the capacity requirements (concurrent connections and performance requirements) increase.

1.1Document Purpose

The purpose of this document is to provide information on the LoadMaster clustering feature and step-by-step instructions on how to set up and configure clustering.

1.2Intended Audience

This document is intended to be read by anyone who is interested in finding out more about LoadMaster clustering.

2LoadMaster Clustering

2.1Prerequisites

There are some prerequisites to be aware of before clustering LoadMasters:

  • A minimum of three LoadMasters are needed to set up clustering. However, four LoadMasters are recommended.
  • All of the LoadMasters in the cluster must be physically identical in terms of the hardware used. Clustering also works with Virtual LoadMasters but the Virtual Machines must have the same resources assigned; this includes CPU, memory and Hard Disk Drive (HDD). Once an empty machine is brought into the cluster, all parameters from the other machines (network interfaces, and so on), are copied over onto the new machine. Local addresses for network interfaces must then be set.
  • Requirements for clustering in a virtual environment (VMware, Hyper-V, and so on) are the same as those for High Availability (HA). In these environments:

Ensure that Media Access Control (MAC) spoofing is allowed and enabled

Ensure that multicast traffic flow is working in both directions between the devices

2.2Restrictions

Most functionality that is available in a single LoadMaster is available when a LoadMaster environment is clustered. The only restrictions are listed below:

  • All Virtual Services must operate at Layer 7
  • Transparency cannot be enabled
  • Subnet originating addresses are always enabled
  • Non-local Real Servers are not supported
  • HA mode cannot be used
  • VXLAN and IPsec tunneling are not supported

2.3Clustering Overview

Figure 2‑1: LoadMaster clustering - example architecture

With LoadMaster clustering, the load-balancing capability can be extended as needed by adding additional LoadMasters for the same Virtual Service. Traffic which is intended for a failed node is either passed on to an existing node or load-balanced across the remaining nodes.

The LoadMasters all work in parallel. If a LoadMaster node fails, traffic is redistributed amongst the remaining nodes. Health checking is performed on each of the LoadMasters.

Most administrative changes should be made on the shared IP address interface. These changes are then propagated to the LoadMasters in less than a second. Local administration of the individual LoadMasters can also be performed by accessing the WUI of the units directly. The WUI of the local LoadMasters contains limited configuration options.

2.4Configuring Clustering

If clustering is enabled on the LoadMaster, the Cluster Control menu item is available under System Configuration in the main menu of the LoadMaster WUI. If clustering functionality is not enabled and you would like to use it, please contact your KEMP representative.

2.4.1Create a New Cluster and Add Nodes to it

To set up a cluster and add LoadMaster nodes to it, follow the steps in the sections below, in order.

2.4.1.1Create a New Cluster

To create a new cluster, follow the steps below on a LoadMaster WUI:

  1. In the main menu of the LoadMaster WUI, go to System Configuration > HA and Clustering.

Figure 2‑2: HA Mode or Clustering

  1. This screen describes both HA Mode and Clustering. Select Clustering and click Confirm.

Figure 2‑3: Convert to Cluster

  1. Click Create New Cluster.

Figure 2‑4: Creating a New Cluster

  1. Enter the desired shared IP address for the cluster.

This is the address that is used to access the shared WUI where configuration changes should be made.

  1. Click Create a New Cluster.

Figure 2‑5: Rebooting

  1. The LoadMaster will then reboot. A prompt will appear asking to reconnect to the shared IP address.
  1. Click Continue.

When the LoadMaster has rebooted, you are brought to the shared IP address.

The cluster is now set up. The next step is to add a LoadMaster to the cluster.

2.4.1.2Add a Node LoadMaster to the Cluster

To add a node LoadMaster to the cluster, the node LoadMaster needs to become available and then have its IP address added in the shared IP address WUI. To do this, follow the steps below:

  1. In the node WUI, go to System Configuration > Cluster Control in the main menu.

Figure 2‑6: Add to Cluster

  1. Click Add to Cluster.

Figure 2‑7: Waiting to Join Cluster

The LoadMaster is now available to be added to the cluster.

  1. Open the WUI of the shared IP address.
  2. In the main menu, go to System Configuration > Cluster Control.

Figure 2‑8: Add New Node

  1. Enter the IP address of the node LoadMaster in the Address text box.
  2. Click Add New Node.

This must be done while the node LoadMaster is available to join the cluster.

Figure 2‑9: Reboot

  1. The node LoadMaster needs to reboot to finish the conversion to cluster mode. Click Continue.

Figure 2‑10: Enable node

  1. By default, the new node is disabled. Click Enable to enable the node in the cluster.

When a node is enabled, it will not be immediately brought into rotation. It will only come online after it has been up for 30 seconds. The page may need to be refreshed to display any state changes. If there are no Virtual Services in the node, the node is in a Disabled state.

  1. Click OK.

The cluster is now configured and a node has been added. To add more nodes, repeat the steps in Section 2.4.1.2.

When all nodes have been added to the cluster, other configuration changes can be made - such as creating and configuring Virtual Services. These changes are propagated from the shared IP address to the LoadMaster nodes.

2.4.1.3Configure the Network Interface(s)

Some configuration changes are needed on the interfaces. Refer to the sections below for further details.

2.4.1.3.1Set the Cluster Shared IP Address

The cluster shared IP address needs to be set on any interfaces that are used to handle traffic. To do this, follow the steps below:

  1. In the main menu of the shared IP address LoadMaster WUI, go to System Configuration > Network Setup.
  1. Click the relevant interface link.

Figure 2‑11: Network Interface Management

  1. Enter the Cluster Shared IP address and click Set Shared address.
2.4.1.3.2Enable the Use for Cluster Checks Option

The Use for Cluster checks option must be enabled on at least one interface. To enable this option, follow the steps below:

  1. In the main menu of the LoadMaster WUI, expand System Configuration.
  1. Click the relevant interface.

Figure 2‑12: Network Interface Management

  1. Tick the Use for Cluster Checks check box.
2.4.1.3.3Enable the Use for Default Gateway Option

The Use for Default Gateway option needs to be enabled on each of the LoadMasters.

To do this, follow the steps below:

  1. In the main menu of a node LoadMaster, expand System Configuration.
  2. Click the relevant interface.

Figure 2‑13: Network Interface Management

  1. Tick the Use for Default Gateway check box.

2.4.2Remove a Node LoadMaster from the Cluster

It is possible to remove a node LoadMaster from the cluster which will mark the node as unavailable for new traffic (that is, a Draining state) to facilitate the removal from the cluster. To do this, follow the steps below in the shared IP address WUI:

  1. In the main menu, go to System Configuration > Cluster Control.

Figure 2‑14: Cluster Control

  1. Click Disable on the node to be removed.

Before a node can be deleted, it must first be disabled. The node must also be online. A node cannot be deleted if it is handling traffic.

  1. The node will change to a Draining state. Wait for the node to finish draining and change to a Disabled state.

The page will need to be refreshed to see the new state.

When a node is in the Draining state, the connections that are still being served by the node are allowed to continue for the amount of time specified in the Node Drain Time text box. For more information, refer to Section 2.6.2.1. No new connections are handled by the node during this time.

  1. Click Delete to remove the node from the cluster.
  2. Click OK to confirm the removal.
  3. Click OK to the confirmation message.

When a node is deleted it becomes a regular single LoadMaster instance. If the LoadMaster is later added back in to the cluster, any configuration changes that have been made in the shared IP address will propagate to the node LoadMaster.

2.4.3Changing the Cluster Parameters

The cluster Virtual ID and node drain time can be updated, if needed. Refer to the sections below for further information.

2.4.3.1Updating the Cluster Virtual ID

When using multiple clusters or LoadMaster HA systems on the same network, the virtual ID identifies each cluster so that there are no potential unwanted interactions. To change the cluster ID, follow the steps below in the shared IP address WUI:

  1. In the main menu, go to System Configuration > Cluster Control.
  1. Click the Show Options button.

Table 2‑1: Cluster Virtual ID

  1. Change the Cluster Virtual ID value.

Valid values range from 1 to 255.

  1. Click Set Cluster Virtual ID.
2.4.3.2Updating the Node Drain Time

When a node is disabled, the connections that are still being served by the node are allowed to continue for the amount of seconds specified in the Node Drain Time text box. No new connections are handled by the node during this time. The Node Drain Time is set to 10 seconds by default. To change the Node Drain Time, follow the steps below in the shared IP address WUI:

  1. In the main menu, go to System Configuration > Cluster Control.
  1. Click the Show Options button.

Table 2‑2: Set Node Drain Time

  1. Change the value in the Node Drain Time text box.

Valid values range between 1 and 600 (seconds).

  1. Click Set Node Drain Time.

2.4.4Backing Up and Restoring the Configuration

Backups can be taken and restored as usual in the System Configuration > System Administration > Backup/Restore section of the LoadMaster WUI. Backups and restores should only be performed on the shared IP address because the changes made there are propagated to the other nodes. When restoring, the individual LoadMasters are rebooted, one at a time.

There are certain settings that need to be set in the Virtual Services for them to work with clustering:

  • Transparency must be disabled
  • Subnet Originating Requests must be enabled
  • All Virtual Services must operate at Layer 7.
  • High Availability (HA) mode cannot be used

As a result of these restrictions, there may be issues with restoring a backup taken from a non-cluster LoadMaster on to a cluster LoadMaster and vice versa, because the settings may be incompatible.

2.5Cluster-Wide Software Updates

The firmware on all LoadMasters in a cluster can be updated using the shared IP address.

Software updates require the LoadMasters to be rebooted. However, the nodes can be rebooted one by one so there is no down time.

To perform a cluster-wide software update, follow the steps below in the WUI of the shared IP address:

  1. Get a LoadMaster firmware patch file.

Visit support.kemptechnologies.com or Contact KEMP Support to get a LoadMaster firmware patch.

  1. In the main menu, go to System Configuration > Update Software.

Table 2‑3: Update Cluster

  1. Click Choose File.
  2. Browse to and select the firmware update patch file.
  3. Click Update Cluster.
  4. Click OK to continue.

Table 2‑4: Validating software update

  1. Wait for the software update to be transferred and validated.

Table 2‑5: Software Update Validated

  1. Click OK to install the software update.
  2. A message will appear saying that the new software has been installed and the Cluster Control screen is displayed.
  3. Each of the non-admin nodes need to be rebooted in order to activate the software. Click Reboot to reboot a node.
  4. Click OK to the pop-up message.
  5. When all non-admin nodes have been rebooted, the admin node needs to be rebooted. Click Reboot in the admin node row to reboot the admin node.

2.6Clustering WUI Options

2.6.1General WUI Options

Figure 2‑15: Cluster Control

If clustering functionality is enabled on a non-node LoadMaster, a Cluster Control menu option should be available in the WUI within the System Configuration section.

Figure 2‑16: Convert to Cluster

Before setting up clustering, clicking the Cluster Control menu item will give the option to either create a new cluster or add this LoadMaster to a cluster.

Figure 2‑17: Local IP menu Figure 2‑18: Shared IP menu

When logging in to the cluster, use the shared IP address to view and set the full functionality of the cluster nodes. Logging in to the local IP address of any of the nodes displays different menu options (see menus above). Logging in to one of the nodes directly is usually reserved for maintenance.

2.6.2Cluster Control WUI Options

Figure 2‑19: Cluster Options

Create New Cluster: If setting up a new cluster, click this button.

Add to Cluster: Add this LoadMaster to an already existing cluster.

Figure 2‑20: Creating a New Cluster

When the Create New Cluster button is clicked, the screen above will appear which prompts to set the shared IP address of the cluster. The shared IP address is the address which is used to administer the cluster.

Figure 2‑21: Rebooting

When the Create a Cluster button is clicked, the LoadMaster reboots. A message will appear asking to reconnect to the shared IP address that was just set.

Figure 2‑22: Cluster Control

After creating a cluster, the Cluster Control screen in the WUI of the shared IP address will allow the addition of LoadMaster nodes into the cluster.

A LoadMaster can only be added to a cluster when the cluster is available and waiting to join the cluster. Refer to Section 2.4.1.2for further information and steps.

Figure 2‑23: Cluster Control

The Cluster Control screen, in the shared IP address WUI, displays details for each of the nodes in the cluster.

Show Options: Clicking the Show Options button will display the Cluster Parameters section which contains two additional fields which can be used to set the Cluster Virtual ID and Node Drain Time. For further information, refer to Section 2.6.2.1.

ID: The cluster ID.

Address: The IP address of the LoadMaster node. If a second IP address appears in brackets after the first one - the second IP address is the IP address of the interface port. The IP address and status text is coloured depending on the status:

  • Blue: The node is the master node.
  • Yellow: The node is disabled.
  • Green: The node is up.
  • Red: The node is down.

Status: The status of the node. The possible statuses are:

  • Admin: The node is the primary control node.
  • Up: The node is up.
  • Down: The node is down.
  • Drain stopping: The node has been disabled and the connections are being shut down in an orderly fashion. Drain stopping lasts for 10 seconds by default. This can be updated by changing the Node Drain Time value on the Cluster Control screen. For more information, refer to Section 2.4.3.2.
  • Starting: The node is starting
  • Disabled: The node is disabled - connections will not be sent to that node. If there are no Virtual Services in the node, the node is in a Disabled state.

Operation: The different operations that can be performed in relation to the notes:

  • Add new node: Add a new node with the specified IP address to the cluster.
  • Disable: Disable the node. Nodes that are disabled will first go through drain stopping. During the drain stopping time, the connections are shut down in an orderly fashion. After the drain, the node is disabled and no traffic is directed to that node.
  • Enable: Enable the node. When a node comes up, it will not be immediately be brought into rotation. It will only come online after it has been up for 30 seconds.
  • Delete: Delete a node from the cluster. When a node is deleted it becomes a regular single LoadMaster instance. If the LoadMaster is later added back in to the cluster, any configuration changes that have been made in the shared IP address will propagate to the node LoadMaster.
2.6.2.1Cluster Parameters

Figure 2‑24: Cluster Parameters

When the Show Options button is clicked, the Cluster Parameters section appears. This section contains two additional WUI options - Cluster Virtual ID and Node Drain Time.

Cluster Virtual ID

When using multiple clusters or LoadMaster HA systems on the same network, the virtual ID identifies each cluster so that there are no potential unwanted interactions. The cluster virtual ID is set to 1 by default, but it can be changed if required. Valid IDs range from 1 to 255. Changes made to an admin Loadmaster propagate across all nodes in the cluster.

Node Drain Time

When a node is disabled, the connections that are still being served by the node are allowed to continue for the amount of seconds specified in the Node Drain Time text box. No new connections are handled by the node during this time. The Node Drain Time is set to 10 seconds by default, but it can be changed if required. Valid values range from 1 to 600 (seconds).

During the drain time the status changes to Draining until the specified drain time elapses.

When the drain time has elapsed the status changes to disabled.

2.6.3Cluster Interface Options

There are a couple of fields relating to clustering in the Network Interface management screen.

Figure 2‑25: Network Interface Management

To get to this screen, go to System Configuration > Network Setup in the main menu of the LoadMaster WUI and click one of the interfaces. The cluster-related options are described below.

Cluster Shared IP address

Specify the shared IP address which can be used to access the cluster. This is also used as the default source address when using Server NAT.

Use for Cluster checks

Use this option to enable cluster health checking between the admin Loadmaster and the nodes. At least one interface must be enabled.

Use for Cluster Updates

Use this interface for cluster synchronization operations.

2.6.4Cluster-Wide Software Update WUI Options

Table 2‑6: Update Cluster

The firmware on all LoadMasters in a cluster can be updated using the shared IP address.

This can be done in System Configuration > System Administration > Update Software by using the Update Cluster option.

After the firmware update patch has been uploaded, a new Reboot button will appear on the Cluster Control screen. When you click Reboot, a pop-up message will appear saying that the node is rebooting.

For more information, including step-by-step instructions on how to perform a cluster-wide software update, refer to Section 2.5.

2.7Application Program Interface (API) Commands

KEMP supports three APIs that can be used to allow remote applications to access the LoadMaster in a simple and consistent manner:

  • RESTful API
  • PowerShell API
  • Java API

Clustering can be configured using API commands. For details on each of the commands that can be used, refer to the relevant API Interface Description document on the KEMP documentation page: www.kemptechnologies.com/documentation.

2.8Troubleshooting

Logs are recorded to track certain clustering events, such as when a node goes offline. Viewing the logs can help to troubleshoot problems which may occur. To view the logs relating to clustering, in the LoadMaster WUI go to System Configuration > Logging Options > System Log Files. The logs relating to clustering are available in the Warning Message File and the System Message File.

Some examples of log entries relating to clustering are listed and described below.

The following log example shows a node failing and another node becoming the main admin node.

lb100 ucarp[2994]: [WARNING]

Switching to state: MASTER

lb100 ucarp[21751]: [CRITICAL]

Partner has failed, becoming

MASTER

ucarp[2995]: [WARNING]

Switching to state: MASTER

ucarp[2995]: [WARNING]

Preferred master advertised:

going back to BACKUP state

ucarp[2995]: [CRITICAL] Partner

has failed, becoming MASTER

lb100 l4d: HA state changed:

now master.

The following log example shows a node going offline. This could be because the node was rebooted.

lb100 l4d: Node 3 now dead

lb100 l4d: Cluster running on

only 3 of 4 nodes

The following log example indicates that a node has gone offline.

lb100 l4d: Cluster running on 3

of 4 nodes (1 disabled)The following log example indicates that a node is alive, for example a disabled node has been enabled.

lb100 l4d: Node 3 now alive

References

Unless otherwise specified, the following documents can be found at http://kemptechnologies.com/documentation.

PowerShell, Interface Description High Availability (HA), Feature Description

Document History

Date

Change

Reason for Change

Version

Resp.

Apr 2015

Initial draft

First draft of document

1.0

LB

June 2015

Changes to reflect beta

Beta version changes

1.1

LB

Aug 2015

New sections added

API steps added

3.0

LB

Sep 2015

Release updates

Updates for 7.1-30

4.0

LB

Oct 2015

Release updates

Updated screenshots

5.0

LB

Dec 2015

Release updates

Updates for 7.1-32

6.0

LB

Jan 2016

Minor updates

Updated

7.0

LB

Mar 2016

Minor updates

Enhancements made

8.0

LB

July 2016

Release updates

Updated for 7.1.35

9.0

LB

Oct 2016

Minor updates

Enhancements made

10.0

LB

Was this article helpful?

0 out of 0 found this helpful

Comments