I am trying to set up a small footprint XenServer lab environment that does not have any single point of failure. That means two hosts in a pool, two NICs on every host and SR (storage repository), multiple paths between hosts and SR, and raid on the SR. The hosts are mini-PCs with only one PCI NIC. XenServer has been made for hardware with several PCI NICs, but with our setup, we will need to use USB NICs. That represents a few challenges.

In fact, USB NICs present so many challenges that they should not be used for production systems. Challenges include difficulties in setting up the USB NICs, reordering such NICs after installation, and the setup getting overwritten by XenServer updates (and possibly patches).

Contents

• Introduction
• Fixed eth1 Address For USB NICs
• Stop Dynamic Naming
• Establish Static Network Rules
• Forget Unwanted NIC Names
• Conclusions
• Figures
• Listings
• Changelog
• References

Introduction

I am struggling with the NICs, because the boxes I have available for hosts only have one ethernet connection without any possibilities for additional cards. Thus, a USB NIC (TP-Link UE300 gigabit) has been added to each host, and the setup now looks like this:

One NIC on each host is connected via USB, and these USB NICs are now causing problems, because they change device name upon each reboot, typically being named NIC-{a changing number}-1. This won't do for use in a pool, as one requirement for pool networking is that device names are consistent across host, that is, if host1 has eth0 and eth1, host2 also needs to have eth0 and eth1, and so forth. As it is, in XenCenter, there are a whole bunch of NICs listed with NIC-{a changing number}-1, and the list gets longer with each reboot. Caos!

There is a workaround available in [2], but it will not work in a pool, as it renames the NICs after XS has already named them. Moreover, interface-rename only works with PCI NICs:

[root@samplehost01 ~]# interface-rename -r
ERROR    [2017-10-19 19:30:15] Can't generate current state for interface '{'Driver': 'cdc_ether', 'Permanent MAC': 'D4:6E:0E:06:2A:09', 'Bus Info': 'usb-0000:00:14.0-2', 'BIOS device': {'all_ethN': 'eth1', 'physical': ''}, 'Assigned MAC': 'D4:6E:0E:06:2A:09', 'Firmware version': 'CDC Ethernet Device', 'Driver version': '22-Aug-2005', 'Kernel name': 'side-6973-eth1'}' - Unrecognised PCI address 'usb-0000:00:14.0-2'

A working solution was proposed by Alan Lantz as set out in [3] and the following write-up.

Fixed eth1 Address For USB NICs

To secure a fixed address for the extra USB NIC on each host, we need to do the following:

• Stop dynamic naming of NICs at boot.
• Establish static network rules.
• Make XenServer forget already registered NIC names that do not correspond to the names we want for the NICs.

Stop Dynamic Naming

Add net.ifnames=0 to the boot. Edit grub in /boot/grub/grub.cfg with either your favourite CLI editor, or by hitting e for edit at bootup.

serial --unit=0 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5
net.ifnames=0
menuentry 'XenServer' {
        search --label --set root root-aptphh
        multiboot2 /boot/xen.gz dom0_mem=752M,max:752M watchdog ucode=scan dom0_max_vcpus=2 crashkernel=128M@256M console=vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.4-xen root=LABEL=root-aptphh ro nolvm hpet=disable xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles
        module2 /boot/initrd-4.4-xen.img
}
menuentry 'XenServer (Serial)' {
        search --label --set root root-aptphh
        multiboot2 /boot/xen.gz com1=115200,8n1 console=com1,vga dom0_mem=752M,max:752M watchdog ucode=scan dom0_max_vcpus=2 crashkernel=128M@256M
        module2 /boot/vmlinuz-4.4-xen root=LABEL=root-aptphh ro nolvm hpet=disable console=tty0 xencons=hvc console=hvc0
        module2 /boot/initrd-4.4-xen.img
}

Establish Static Network Rules

Any persistent rules in 70-persistent-net.rules will thus be overruled by xen-backend.rules, Instead, create a new rule called 71-persistent-net.rules.

 $ sudo vi /etc/udev/rules.d/71-persistent-net.rules
	
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="D4:6E:0E:06:2A:09", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

We now need to reboot to apply the naming rule for eth1. Just enter reboot in the CLI to reboot.

We also need to define a static IP-address for the NIC, in order for it to work with the pool, by using xe pif-reconfigure:

xe pif-reconfigure-ip uuid=4df1ed96-e208-1182-44ee-3a9bea79c1d4 mode=static IP=10.0.0.54 netmask=255.255.255.0 gateway=10.0.0.1 DNS=8.8.8.8

Forget Unwanted NIC Names

After completing the steps above, we got two named NICs on the same MAC, one "side-xxx-1" and one eth1:

[root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label
uuid ( RO): 07935775-8be8-6ce7-9551-edddb81a9c06
device ( RO): side-8373-eth1
host-name-label ( RO): xenserver-sampleserver02

uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4
device ( RO): eth1
host-name-label ( RO): xenserver-sampleserver02

uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a
device ( RO): eth0
host-name-label ( RO): xenserver-sampleserver02

We need to employ xe pif-forget to get rid of the "side-NIC" name. By using xe pif-forget, we make sure that the "side-NIC" will be consistently removed throughout reboots:

[root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label
uuid ( RO): 07935775-8be8-6ce7-9551-edddb81a9c06
device ( RO): side-8373-eth1
host-name-label ( RO): xenserver-sampleserver02

uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4
device ( RO): eth1
host-name-label ( RO): xenserver-sampleserver02

uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a
device ( RO): eth0
host-name-label ( RO): xenserver-sampleserver02

[root@xenserver-sampleserver02 ~]# xe pif-forget uuid=07935775-8be8-6ce7-9551-edddb81a9c06
[root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label

uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4
device ( RO): eth1
host-name-label ( RO): xenserver-sampleserver02

uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a
device ( RO): eth0
host-name-label ( RO): xenserver-sampleserver02

Finally, let's check that the network setup is correct:

[root@xenserver-sampleserver02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 2c:56:dc:ba:f3:e9 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN qlen 1000
    link/ether d4:6e:0e:06:e9:36 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1
    link/ether 46:89:64:8f:a7:a9 brd ff:ff:ff:ff:ff:ff
5: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1
    link/ether 2c:56:dc:ba:f3:e9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.53/24 brd 10.0.0.255 scope global xenbr0
       valid_lft forever preferred_lft forever
6: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1
    link/ether d4:6e:0e:06:e9:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.54/24 brd 10.0.0.255 scope global xenbr1
       valid_lft forever preferred_lft forever

You can also get an overview of the network setup in XenCenter or in the xsconsole via the CLI:

We got two working NICs, eth0 and eth1, both with a static network setup. All is well.

Conclusions

We can avoid dynamic addition of NIC-names for USB NICs by adding net.ifnames=0 to grub in /boot/grub/grub.cfg, setting a static network configuration with xe pif-reconfigure, and forgetting unwanted NIC-names with xe pif-forget.

Changes to grub.cfg may be overwritten by XenServer updates and patches. You have to check if a patch or an update has affected the setup.

Figures

• Figure 1 - XenServer multipath setup with one NIC via USB
• Figure 2 - XenServer xsconsole network overview

Listings

• Listing 1 - interface-rename-command error
• Listing 2 - net.ifnames=0 in grub.cfg>
• Listing 3 - Edit persisten network rules
• Listing 4 - Setting a static network confguration with xe pif-reconfigure.
• Listing 5 - "side-xxx-1" NIC-name
• Listing 6 - Forgetting a NIC-name with xe pif-forget.
• Listing 7 - Listing the network status with ip a.

Changelog

• 2017.11.03 - Added that a reboot is required after editing the renaming rules for eth1.
• 2017.11.21 - Added warning at the beginning of the article.

References

• [1]How to Change Order of NICs in XenServer, Downloaded October 24, 2017 - https://support.citrix.com/article/CTX135809
• [2]USB NIC with XenServer 7?, Downloaded October 24, 2017 - https://discussions.citrix.com/topic/379716-usb-nic-with-xenserver-7/
• [3]USB NICs In XS Pool Feasible?Published October 19, 2017 - https://discussions.citrix.com/topic/390537-usb-nics-in-xs-pool-feasible/