When adding 4 or more VMXNET3 NICs to a VLM in VMware, the order is incorrect
When adding 4 or more VMXNET3 Network Interface Controllers (NICs) to a Virtual LoadMaster (VLM) in VMware, the order of the interfaces on the Web User Interface (WUI) does not correspond with the order of the interfaces in machine setting, for example:
NIC 1: eth2
NIC 2: eth1
NIC 3: eth3
NIC 4: eth0
This order has been observed in all cases. It does not matter if the NICs are added before licensing or if they are added later.
eth0 always corresponds to NIC4.
This behaviour does not affect e1000 NICs.
This is a VMware issue and nothing to do with the KEMP LoadMaster. For further information, please refer to the following articles:
https://communities.vmware.com/thread/443600
Comments
@software
Double checked myself on Vsphere 6.5 and this is still the case. Unfortunately this is a VMware issue and I could not find any new relevant articles on this.
I have VMware 6.7, its Aug 2019 and this issue is still there. I actually cannot add more than 4 NICS to the LB.
@rwashburn
We understand your frustration with this issue, but as previously stated in the article, this is a VMware issue that they have not addressed. Based on the SR you submitted, another option would be configuring a trunk on the VMware side and creating the VLANS on the LM. This would require less NICs on the LM and our team would be happy to assist in getting this configured.
Hi Kemp. One way to work around this issue for customers is to include (10) vmxnet3 nics (the max number) in the KEMP virtual appliance, then modify the vmx file for each "ethernet#.pciSlotNumber = " line, and specify the correct order of the slot number to match it up with so it's one to one.
# Here's the default mapping logic from VMware:
# Mapping PCI slot numbers to guest-visible PCI bus topology (2047927)
# https://kb.vmware.com/s/article/2047927
# As a test I made a test VM, and made note of each ethernet#.pciSlotNumber and pciBridge#.pciSlotNumber.
# According to that VMware article, I got the below. (Note, I haven't tested any of the below.) I did notice that it does match to the answer in the original post, but in reverse order.
# FFF.BBBBB.DDDDD # BBBBB-1 # Bus:(pciBridge#.pciSlotNumber(DDDDD)).(ethernet#.pciSlotNumber(FFF))
ethernet0.pciSlotNumber = "1184" # 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0
ethernet1.pciSlotNumber = "1216" # 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3
ethernet2.pciSlotNumber = "1248" # 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6
ethernet3.pciSlotNumber = "1280" # 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8
ethernet4.pciSlotNumber = "2208" # 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet5.pciSlotNumber = "2240" # 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4
ethernet6.pciSlotNumber = "2272" # 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7
ethernet7.pciSlotNumber = "2304" # 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9
ethernet8.pciSlotNumber = "3232" # 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2
ethernet9.pciSlotNumber = "3264" # 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5
It appears the VMware enumeration order is Dev increments, then Func (as in the parent slot increments first, then the device slot).
Whereas I believe the Guest OS is expecting the order to increase by Func, then Dev (as in the device increments first, then the parent slot).
For example, such as using the pciSlotNumbers below instead might make them match up. (If you get an error, then try the reverse order, as in eth0 to 2304, and eth9 to 1184).
ethernet0.pciSlotNumber = "1184" # 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0
ethernet1.pciSlotNumber = "2208" # 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet2.pciSlotNumber = "3232" # 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2
ethernet3.pciSlotNumber = "1216" # 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3
ethernet4.pciSlotNumber = "2240" # 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4
ethernet5.pciSlotNumber = "3264" # 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5
ethernet6.pciSlotNumber = "1248" # 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6
ethernet7.pciSlotNumber = "2272" # 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7
ethernet8.pciSlotNumber = "1280" # 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8
ethernet9.pciSlotNumber = "2304" # 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9
# I haven't tested any of the above.
# However, I imagine that VMware enumerate their default order to spread out the usage of the parent pci slots, for performance reasons. For example, 3 virtual devices on 3 separate virtual parent slots may have better performance, than 3 virtual devices on a single virtual parent slot. I know I've seen similar VMware recommendations, such as for using multiple VMware disks and using multiple PVSCSI controllers, instead of a single PVSCSI controller.
Hardcoding the ethernet#.pciSlotNumber for all 10 possible vmxnic3 nics would avoid this issue for your customers. By default, I'd recommend adding the max vmxnet3 nics which is 10, but only enabling 1 or 2, and let the customer enable the rest of the vmxnet3 nics if needed.
V/r,
Darren DeHaven
Butha van der Merwe
A date on this article would be of great help - as the VM article referenced dates back to 2013 - so no confirmation if this issue still exists today in 2018 with vsphere 6.5 U1