Kemp Support, how can we help?

The latest application delivery knowledge and expertise at your fingertips.

  1. Kemp Support
  2. Knowledge Base
  3. Platforms

When adding 4 or more VMXNET3 NICs to a VLM in VMware, the order is incorrect

Updated: 

When adding 4 or more VMXNET3 Network Interface Controllers (NICs) to a Virtual LoadMaster (VLM) in VMware, the order of the interfaces on the Web User Interface (WUI) does not correspond with the order of the interfaces in machine setting, for example:

NIC 1: eth2
NIC 2: eth1
NIC 3: eth3
NIC 4: eth0

This order has been observed in all cases. It does not matter if the NICs are added before licensing or if they are added later.

eth0 always corresponds to NIC4.

This behaviour does not affect e1000 NICs.

This is a VMware issue and nothing to do with the KEMP LoadMaster. For further information, please refer to the following articles:

https://communities.vmware.com/thread/443600

http://vnetwise.wordpress.com/2013/03/22/howto-force-vmware-esxi-to-re-order-nic-ports-to-maintain-standards/

 

Comments

Avatar

Butha van der Merwe

A date on this article would be of great help - as the VM article referenced dates back to 2013 - so no confirmation if this issue still exists today in 2018 with vsphere 6.5 U1

0
Avatar

Nick Smylie

@software

Double checked myself on Vsphere 6.5 and this is still the case. Unfortunately this is a VMware issue and I could not find any new relevant articles on this.

0
Avatar

rwashburn

I have VMware 6.7, its Aug 2019 and this issue is still there. I actually cannot add more than 4 NICS to the LB.

0
Avatar

Justin Federico

@rwashburn

We understand your frustration with this issue, but as previously stated in the article, this is a VMware issue that they have not addressed. Based on the SR you submitted, another option would be configuring a trunk on the VMware side and creating the VLANS on the LM. This would require less NICs on the LM and our team would be happy to assist in getting this configured.

0
Avatar

System Administration

Hi Kemp. One way to work around this issue for customers is to include (10) vmxnet3 nics (the max number) in the KEMP virtual appliance, then modify the vmx file for each "ethernet#.pciSlotNumber = " line, and specify the correct order of the slot number to match it up with so it's one to one.

# Here's the default mapping logic from VMware:
# Mapping PCI slot numbers to guest-visible PCI bus topology (2047927)
# https://kb.vmware.com/s/article/2047927

# As a test I made a test VM, and made note of each ethernet#.pciSlotNumber and pciBridge#.pciSlotNumber.

# According to that VMware article, I got the below. (Note, I haven't tested any of the below.) I did notice that it does match to the answer in the original post, but in reverse order.
# FFF.BBBBB.DDDDD # BBBBB-1 # Bus:(pciBridge#.pciSlotNumber(DDDDD)).(ethernet#.pciSlotNumber(FFF))
ethernet0.pciSlotNumber = "1184" # 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0
ethernet1.pciSlotNumber = "1216" # 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3
ethernet2.pciSlotNumber = "1248" # 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6
ethernet3.pciSlotNumber = "1280" # 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8
ethernet4.pciSlotNumber = "2208" # 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet5.pciSlotNumber = "2240" # 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4
ethernet6.pciSlotNumber = "2272" # 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7
ethernet7.pciSlotNumber = "2304" # 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9
ethernet8.pciSlotNumber = "3232" # 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2
ethernet9.pciSlotNumber = "3264" # 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5

It appears the VMware enumeration order is Dev increments, then Func (as in the parent slot increments first, then the device slot).
Whereas I believe the Guest OS is expecting the order to increase by Func, then Dev (as in the device increments first, then the parent slot).

For example, such as using the pciSlotNumbers below instead might make them match up. (If you get an error, then try the reverse order, as in eth0 to 2304, and eth9 to 1184).
ethernet0.pciSlotNumber = "1184" # 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0
ethernet1.pciSlotNumber = "2208" # 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet2.pciSlotNumber = "3232" # 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2
ethernet3.pciSlotNumber = "1216" # 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3
ethernet4.pciSlotNumber = "2240" # 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4
ethernet5.pciSlotNumber = "3264" # 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5
ethernet6.pciSlotNumber = "1248" # 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6
ethernet7.pciSlotNumber = "2272" # 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7
ethernet8.pciSlotNumber = "1280" # 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8
ethernet9.pciSlotNumber = "2304" # 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9

# I haven't tested any of the above.
# However, I imagine that VMware enumerate their default order to spread out the usage of the parent pci slots, for performance reasons.  For example, 3 virtual devices on 3 separate virtual parent slots may have better performance, than 3 virtual devices on a single virtual parent slot.  I know I've seen similar VMware recommendations, such as for using multiple VMware disks and using multiple PVSCSI controllers, instead of a single PVSCSI controller.

Hardcoding the ethernet#.pciSlotNumber for all 10 possible vmxnic3 nics would avoid this issue for your customers.  By default, I'd recommend adding the max vmxnet3 nics which is 10, but only enabling 1 or 2, and let the customer enable the rest of the vmxnet3 nics if needed.

 

V/r,

Darren DeHaven

 

0
Avatar

Darren DeHaven

It would also be nice if the WUI included the MAC address of the interface.

Another possible way to avoid this issue from a customer side, is to have each network connection configured the same for each uplink with all the vlan tags to be used by the kemp node, but specify whichever vlan tag to be used on each interface within kemp.

0
Avatar

Michael Starr

Hello,

 

The information you are looking for is in our debug options under Ifconfig. This gives the mac address and Vlan information. Although having the Mac address located on the interface would be useful. You can suggest that on our feature request page located at https://ideas.kemp.ax

 

 

 

0
Avatar

Darren DeHaven

Thanks Mike.

I've verified that I could get the eth and vmware nic order to be in sequence, by doing the following:

* Upgraded VMware hardware to latest (probably doesn't matter)

* Added all 10 vmware nics (max vmware nics)

* Downloaded .vmx file, made below changes, replaced it, then reregistered the VM.  

* Commented out any existing "ethernet#.pciSlotNumber" entries to .vmx

* Added below to .vmx:

ethernet0.pciSlotNumber = "1184"

# 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0

ethernet1.pciSlotNumber = "2208"

# 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet2.pciSlotNumber = "3232"

# 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2

ethernet3.pciSlotNumber = "1216"

# 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3

ethernet4.pciSlotNumber = "2240"

# 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4

ethernet5.pciSlotNumber = "3264"

# 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5

ethernet6.pciSlotNumber = "1248"

# 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6

ethernet7.pciSlotNumber = "2272"

# 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7

ethernet8.pciSlotNumber = "1280"

# 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8

ethernet9.pciSlotNumber = "2304"

# 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9

 

0
In this document