When deploying an application using CenturyLink Cloud, you may need to provide load balancing and other application delivery functions such as content switching, SSL acceleration, Web Application Firewall (WAF), and Intrusion Prevention System (IPS). When using a Kemp LoadMaster for CenturyLink, you can address both application delivery and High Availability (HA).
Some of the features and associated benefits of the Virtual LoadMaster (VLM) are listed in the table below.
Regardless of where the applications are deployed (cloud, on premises, or in hybrid environments), the VLM can load balance them.
The VLM manages applications deployed in hybrid infrastructures on premises and in the CenturyLink Cloud.
Highly available Application Delivery Controllers (ADCs), deployed on-demand to meet load requirements.
VLM Global Balancing (GEO) load balancing supports application instances across multiple sites to accommodate growth and deliver additional resilience.
Kemp provides two different approaches for deploying LoadMasters with HA in CenturyLink Cloud: native HA and HA using GEO. The approach to use is determined by the CenturyLink data center that the LoadMasters are deployed in.
CenturyLink native HA is the first approach and it leverages the CenturyLink load balancer to route traffic to a healthy LoadMaster. The CenturyLink load balancer features required for this functionality are available in select data centers. These data centers are marked as advanced capabilities.
The CenturyLink HA using GEO (GSLB) approach leverages the Kemp LoadMaster GEO functionality. This functionality requires DNS delegation and configuration to route the active LoadMaster. This approach does not require the CenturyLink load balancer.
This document is intended to explain the two different configurations of HA for the Kemp LoadMaster in CenturyLink Cloud.
It is also possible to configure the LoadMaster using Application Program Interface (API) commands. For further details, refer to the Interface Description documents on the Kemp documentation page: https://kemptechnologies.com/documentation.
This document is intended to be read by anyone who is interested in deploying a HA pair of LoadMasters in CenturyLink Cloud.
To build a HA LoadMaster environment there are several settings that you must carefully specify. Follow the steps below to set up HA:
1. Log in to the LoadMaster that you want to be the active (master) unit.
2. In the main menu, select System Configuration and click HA Parameters.
3. A screen appears asking if you want to set up HA Mode or Clustering. To set up HA, select HA Mode and click Confirm.
For instructions on configuring clustering, refer to the LoadMaster Clustering Feature Description on the Kemp documentation page.
4. Select HA (First) Mode in the HA Mode drop-down.
5. Click OK.
Do not reboot at this time.
6. Click OK on the resulting message reminding not to forget to set the Shared IP address.
Selecting the Prevent this page from creating additional dialogs check box stops any warning messages, such as this one, from appearing.
7. Specify the desired shared IP address in the HA Shared IP address field and click Set Shared address.
8. A confirmation message may appear. Click OK.
Do not reboot or reconnect at this time.
9. Enter the IP address of the standby unit in the HA Partner IP address field and click Set Partner address.
10. A confirmation message appears. Click OK.
11. As of the 7.2.36 firmware, the LoadMaster selects a HA Virtual ID based on the shared IP address of the first configured interface (the last 8 bits). You can change the value to whatever you want (in the range 10 â 255) or you can keep it at the value it already selected.
Ensure the Virtual ID is unique for each HA pair on the network. When using multiple HA LoadMaster clusters (or other devices using CARP-like protocols) on the same network, this value uniquely identifies each cluster so that there are no potential unwanted interactions.
12. Configure any other settings as needed.
13. Click Reboot Now.
14. Click Continue.
15. Refresh the page after the LoadMaster has rebooted (this may take a few minutes).
A log in screen appears. After logging in, a different menu appears than before. This is the Local Administration menu displayed for HA units. This menu has far fewer options. Only configuration settings pertaining to that specific unit are accessible using the Local Administration option. All management of the HA units should be done using the shared IP address. To see the full menu and configure the units, access the WUI of the shared IP address, which was specified above.
16. Log in to the shared IP Web User Interface (WUI) by entering the shared IP address in the address bar of the browser and pressing Enter.
In the top-right of the screen there are two indicator squares. These squares indicate the status of the HA pair. The left square always represents HA1 and the right represents HA2. The A represents which unit is active. The first or second HA unit can be opened by clicking the relevant status icon. Green and green status colors indicate a properly paired configuration. Currently, the icons are probably green and red since the HA2 unit has not yet joined the pair. For an explanation of all icon colors and statuses, refer to the HA Parameters section.
17. Go to HA Parameters in the main menu.
18. Enter a different number (different from the IDs of other HA devices) in the HA Virtual ID text box and click Set Virtual ID. Using the same ID as other HA devices may cause problems.
All HA pairs on the network must be assigned unique HA Virtual ID numbers.
Now that HA has been configured on the first unit, the second unit must be set up. Follow the steps below to do this:
1. Enter the IP address of the second unit in the address bar of the browser and press Enter.
Ensure to enter https:// before the IP address.
2. In the main menu, select System Configuration and click the HA option.
3. A screen appears asking if you want to set up HA Mode or Clustering. To set up HA, select HA Mode and click Confirm.
4. Select HA (Second) Mode as the HA Mode.
5. Click OK.
6. Click OK.
Ticking the Prevent this page from creating additional dialogs check box stops any warning messages, such as this one, from appearing.
7. Enter the HA Shared IP address and click Set Shared address.
The HA Shared IP address must be the same as the HA Shared IP address which was set when configuring the first unit in the Set up the First Unit section.
8. Click OK.
9. Click OK on the message asking to reconnect to the shared IP address.
10. Enter the IP address of the first (master) unit in the HA pair in the HA Partner IP address field and click Set Partner address.
11. Click OK.
12. Ensure the HA Virtual ID is the same as it is on the other unit.
If they are not the same, the pairing fails.
13. Change any other settings as needed.
14. Click Reboot Now.
15. Click Continue.
Passwords for the bal account are not synchronized across HA pairs, so ensure to use the same password on both units. Problems may occur if different passwords are used.
After rebooting, the HA pair establishes a TCP connection (using port 6973) between the two addresses. The synchronization process is started for the configuration.
The indicator squares should now be green and green. The A indicates the active unit of the pair. If the first synchronization attempt fails (that is, the icons are not green and green) a second attempt might be needed.
On the home screen, the IP address field has changed. In addition to specifying the shared IP address of the pair, it also specifies the IP address of the unit. The left IP address is the shared address. The IP address in parentheses is the address of the current unit.
Some guidelines relating to the Use for HA checks option are below:
- If the Use for HA checks check box is grayed out it means that this is the only interface configured to be used for HA checks and it cannot be deselected.
To enable the Use for HA checks option, follow the steps below:
1. Go to the WUI of the shared IP address.
2. In the main menu, select System Configuration.
3. Select the relevant interface.
4. Select the Use for HA checks check box.
These steps can be repeated if you need to enable the Use for HA checks option on more than one interface.
Now that the HA units have been set up, failover can be tested if needed. The easiest way to do this is to reboot the active unit. To reboot the unit, follow the steps below:
1. Log in to the IP address of the active unit.
2. In the main menu, click Local Administration.
3. Select System Reboot.
4. Click Reboot.
5. A confirmation message may appear. Click OK.
6. Click Continue.
When HA1 is back online, both HA status icons should be green. The A should have moved into the right green square. This means that the secondary unit is now the active unit.
When using local certificates in HA mode â the shared IP inherits the local certificate from the master unit. So, if a standby unit has a different local certificate to the master and failover occurs, the shared IP inherits the local certificate of the standby (now master) unit.
Kemp recommends performing firmware updates outside of working hours. This ensures there is no interruption to client connectivity. If it has to be done during working hours, Kemp recommend scheduling a maintenance window.
Before updating the firmware, ensure the Switch to Preferred Server drop-down list is set to No Preferred Host in System Configuration > HA Parameters.
Kemp recommends updating the passive unit first and then updating the active unit. This causes only a single failover and minimal downtime, and is the preferred option for most customers. While this procedure does leave the currently passive LoadMaster as the active LoadMaster going forward, this usually has no consequences in most customer environments. However, it is also possible to update the currently active unit, failover to the passive unit, update the passive unit and then failover to the originally active unit.
To update the firmware on a HA pair using the recommended method; perform the following steps using the shared IP address:
1. Update the passive LoadMaster first (we will refer to this LoadMaster as B).
2. When the update is complete, reboot B.
3. When unit B is back up, update the active unit (we will refer to this unit as A).
4. When the update is complete, reboot A. Now B becomes active.
5. Ensure B is handling traffic.
See below for descriptions of the various HA-related fields in the LoadMaster WUI.
If the unit is part of a HA configuration, the following screen displays when you click one of the interfaces.
This screen tells the user:
- The IP address of this LoadMaster (10.35.47.10 in this example).
- The HA shared IP address (10.35.47.30 in this example). This is the IP address used to configure the pair.
- The IP address of the paired machine (10.35.47.12 in this example).
- Whether or not this interface is enabled for HA health-checking.
- The speed of the link (automatically detected). If the link is down, it is indicated here.
- Any alternate addresses on this interface.
Some key points to note about this option are below:
- The Use for HA checks check box must be selected on at least one interface that has connectivity from HA1 to HA2.
- If the Use for HA checks check box is grayed out it means that this is the only interface configured to be used for HA checks and cannot be deselected.
- This option should include at least one production interface, because if HA checks are only selected on non-production interfaces, the backup unit does not notice if a production interface goes down and does not take over for the incapacitated unit.
You can change the role of the LoadMaster by setting the HA Mode. If the HA Mode is set to HA (First) Mode or HA (Second) Mode, a prompt appears reminding you to add a shared IP. Changing the HA Mode requires a reboot. After the details are set, click Reboot. Once the LoadMaster has rebooted, the HA Parameters menu option is available in the System Configuration section provided the role is not Non HA Mode. Configuring both units in the same HA Mode, for example, HA (First Mode) and HA (First) Mode, results in severe operational problems because; not only will both units be master, both units try to use the same IP address.
When logging in to the HA cluster, use the shared IP address to view and set the full functionality of the pair, apart from passwords and licensing. Logging in to the direct IP address of either one of the devices displays different menu options (see menus below). Logging into one of the LoadMasters directly is usually reserved for maintenance.
When a LoadMaster is in HA mode, the following screen appears when the HA Parameters menu option is selected:
After initial configuration, the HA parameters should not be modified unless both units in the HA pair are available and operating properly (if they are both showing green icons at the top of the WUI, with one LoadMaster in active mode and the other in standby).
At the top of the screen, next to the time, icons denote the real-time status of the LoadMaster units in the cluster. There is an icon for each unit in the cluster. This status is maintained using an automatic ping between the units.
Clicking these icons opens the management interface of the relevant HA partner.
The possible icons are:
Green (with âAâ)
The unit is online and operational and the HA units are correctly paired.
The A in the middle of the square indicates that this is the master (active) unit.
Green (without âAâ)
The unit is online and operational and the HA units are correctly paired.
The absence of an âAâ in the middle of the square indicates that this is not the master unit (standby).
The partner unit is unreachable or turned off. It may be offline or misconfigured. The unit is not ready to take over. It may be offline or incorrectly paired.
When the unit reboots more than three times in 5 minutes it enters a pacified state. In this state the machine is only accessible using the direct machine WUI (not the shared WUI) and it is not participating in any HA activity. Therefore, no changes from the master are received and it does not take over if the master fails. To remove the unit from the pacified state, fix the root cause of the health check failures, log in to the pacified LoadMaster through SSH or the console and reboot.
If a unit continuously reverts to a pacified state, check the network to see if CARP is being blocked.
The machine is in an indeterminate state and may require a reboot to return to operation. A gray box often means the unit has not been set up in HA mode correctly. A gray box also appears for a few seconds during the initial HA configuration.
In some cases, it may mean both machines are active, that is, both are set to master, and something has gone seriously wrong.
|Question marks||The HA status is updating.|
|Both green (left box with 'A')||Both units are up, unit 1 is master and unit 2 is standby.|
|Both green (right box with 'A')||Both units are up, unit 1 is standby and unit 2 is master.|
|Left box green, right box red/yellow||Unit 1 is up and currently master. Unit 1 cannot reach unit 2, or unit 2 is turned off.|
|Left box red/yellow, right box green||Unit 2 is up and currently master. Unit 2 cannot reach unit 1, or unit 1 is turned off.|
|Left box gray, right box red/yellow||HA setup is not complete on unit 1.|
|Left box red/yellow, right box gray||HA setup is not complete on unit 2.|
|No HA icons||
If the HA status squares are not appearing in the WUI, it probably means that HA is not enabled. Go to System Administration and select the HA option. Ensure the HA Mode is set to either First or Second.
In HA mode, each LoadMaster has its own IP address that is used only for diagnostic purposes directly on the unit. The HA pair have a shared IP address over which the WUI is used to configure and manage the pair as a single entity.
There are a number of prerequisites that must be in place for HA to function correctly. Refer to the Prerequisites section for a list of these prerequisites.
If using a single LoadMaster, select NonHA Mode. When setting up HA mode, one LoadMaster must be set to HA (First) Mode and the other HA (Second) Mode. HA does not operate if both units have the same HA Mode.
CARP requests are sent every second from the master. The value selected in the HA Timeout drop-down list is the time that the master machine must be unavailable before a switchover occurs. With this option, the time it takes a HA cluster to detect a failure can be adjusted from 3 seconds to 15 seconds in 3-second increments. The default value is 9 seconds. A lower value detects failures sooner, whereas a higher value prevents HA from failing over too soon if there is a delay when receiving CARP.
To set this option, follow the steps below:
1. Select System Configuration > HA Parameters.
2. Select the preferred value in the HA Timeout drop-down list.
The HA Initial Wait Time is the length of time after the initial boot of a LoadMaster, before the machine decides that it should become active. If the partner machine is running, this value is ignored. You can change this value to mitigate the time taken for some intelligent switches to detect that the LoadMaster has started and to bring up the link.
When using multiple HA LoadMaster clusters (or other devices using CARP-like protocols) on the same network, this value uniquely identifies each cluster so that there are no potential unwanted interactions.
Kemp highly recommends using a higher value than 10 because any other HA pair using the same ID could interfere with HA operations.
As of the 7.2.36 release, the LoadMaster selects a virtual ID based on the shared IP address of the first configured interface (the last eight bits). It is selected and displayed once both the shared address and the partner address are set. You can change the value to whatever you want (in the range 1 â 255) or you can keep it at the value it already selected. Ensure the virtual ID is unique on each LoadMaster on the network.
You can find the HA Virtual ID in the LoadMaster WUI by going to System Configuration > HA Parameters.
Switch to Preferred Server
By default, neither partner in a HA cluster has priority. When a machine restarts after a failover, the machine becomes the standby and stays in that state until it is forced to master. Specifying a preferred host means that when this machine restarts, it always tries to become master and the partner reverts to standby mode.
When set to Prefer First HA, if the LoadMaster fails over, the master reverts to HA1 when HA1 comes back online.
When set to Prefer Second HA, if the LoadMaster fails over, the master reverts to HA2 when HA2 comes back online.
When No Preferred Host is selected, if there is a failover on the LoadMaster, the unit that becomes master remains as master (failback does not happen).
To change this option, follow the steps below in the LoadMaster WUI:
1. In the main menu, select Local Administration > HA Parameters.
2. Select the relevant option from the Switch to Preferred Server drop-down list.
Some connections may be dropped during the switchover if a preferred host is specified.
For normal operating conditions, Kemp recommends selecting No Preferred Host.
The interface used to synchronize the entire HA configuration within the HA cluster. Synchronization occurs every two minutes. The information is synchronized over SSH port 6973.
Force Partner Update
Immediately forces the configuration from the active to standby unit without waiting for a normal update. This option is only available if both units can see each other in an active/standby scenario.
Enabling this option can have a significant performance impact.
HA Multicast Interface
The network interface used for multicast traffic, which is used to synchronize Layer 4 and Layer 7 traffic when Inter HA Updates are enabled.
You can select the interface to send and receive inter-HA traffic from within the WUI of the shared IP address:
1. In the main menu, select System Configuration > HA Parameters.
2. The HA Update Interface setting is used for sending HA configuration updates using TCP/6973 between units. Modify it if needed.
If you have enabled L7 persistency updates or L4 TCP connection updates, an additional HA Multicast Interface option also becomes available.
This section outlines troubleshooting steps for some common HA-related problems. If further help is needed, contact Kemp Support.
General HA troubleshooting steps are below:
- Check that the IP settings for the Interface, Partner, and Shared IP address are correct. These settings can be found in System Configuration > Interfaces.
- Log in to each of the single HA interface addresses and ensure the HA parameters are correct (Local Administration > HA Parameters):
- Ensure that the HA pair has one unit in HA (First) Mode and another unit in HA (Second) Mode.
- Ensure that both units are on the same protocol and HA ID.
- Ensure that all of the IP addresses are available and are not in use by another device. IP conflict causes numerous problems.
- Shut down one or both LoadMasters and try to ping the IP address of each unit. If there is an answer, another device is using that IP address. Try the âarp -aâ or ânetstatâ commands to find out more information on what device that is.
- Set the HA Virtual ID (in Local Administration > HA Parameters) to something other than 1. The further up the range the better â avoid numbers from 1 to 10 because other HA pairs may have those IDs and use Virtual Router Redundancy Protocol (VRRP).
The Virtual ID can conflict with any device on the network which is using VRRP. If there are multiple HA clusters on the same network, they must also have different Virtual IDs.
- Check that the time of both units are in sync and if they are not, ensure that NTP is configured and running on both units.
- Ensure there are no Virtual Services using TCP and port 6973 on the interface where synchronization is configured.
- Ensure there are no Virtual Services on either of the HA individual addresses.
- Ensure there are no Virtual Services using TCP and port 22 on a LoadMaster interface port.
If you are experiencing problems with HA, confirm that:
- The two LoadMasters are on the same subnet.
- For each Network Interface Card (NIC) that has Use for HA checks enabled, the link status shows as connected.
- The two units can ping each other and their default gateway. (The ping options are available in System Configuration > Logging Options > System Log Files > Debug Options.)
- Both units have the same time. Set both units to use the same NTP server and correct time zone. (The date and time options are available in System Configuration > System Administration > Date/Time).
- Unit 1 is set to HA (First) Mode.
Unit 2 is set to HA (Second) Mode.
The HA Virtual ID is the same on both units.
While the instructions above provide a basic overview of how to deploy a LoadMaster HA Pair in CenturyLink Cloud, it is not designed to be a comprehensive guide. This section identifies some of many guides published on the resources section of our website. Unless otherwise specified, the following documents can be found at http://kemptechnologies.com/loadmaster-documentation.
Licensing, Feature Description
Web User Interface (WUI), Configuration Guide
This document was last updated on 07 August 2019.