I am trying to set up a small footprint XenServer lab environment that does not have any single point of failure. That means two hosts in a pool, two NICs on every host and SR (storage repository), multiple paths between hosts and SR, and raid on the SR. The hosts are mini-PCs with only one PCI NIC. XenServer has been made for hardware with several PCI NICs, but with our setup, we will need to use USB NICs. That represents a few challenges.
In fact, USB NICs present so many challenges that they should not be used for production systems. Challenges include difficulties in setting up the USB NICs, reordering such NICs after installation, and the setup getting overwritten by XenServer updates (and possibly patches).
Contents
- Introduction
- Fixed eth1 Address For USB NICs
- Stop Dynamic Naming
- Establish Static Network Rules
- Forget Unwanted NIC Names
- Conclusions
- Figures
- Listings
- Changelog
- References
Introduction
I am struggling with the NICs, because the boxes I have available for hosts only have one ethernet connection without any possibilities for additional cards. Thus, a USB NIC (TP-Link UE300 gigabit) has been added to each host, and the setup now looks like this:

One NIC on each host is connected via USB, and these USB NICs are now causing problems, because they change device name upon each reboot, typically being named NIC-{a changing number}-1. This won't do for use in a pool, as one requirement for pool networking is that device names are consistent across host, that is, if host1 has eth0 and eth1, host2 also needs to have eth0 and eth1, and so forth. As it is, in XenCenter, there are a whole bunch of NICs listed with NIC-{a changing number}-1, and the list gets longer with each reboot. Caos!
There is a workaround available in [2], but it will not work in a pool, as it renames the NICs after XS has already named them. Moreover, interface-rename
only works with PCI NICs:
[root@samplehost01 ~]# interface-rename -r ERROR [2017-10-19 19:30:15] Can't generate current state for interface '{'Driver': 'cdc_ether', 'Permanent MAC': 'D4:6E:0E:06:2A:09', 'Bus Info': 'usb-0000:00:14.0-2', 'BIOS device': {'all_ethN': 'eth1', 'physical': ''}, 'Assigned MAC': 'D4:6E:0E:06:2A:09', 'Firmware version': 'CDC Ethernet Device', 'Driver version': '22-Aug-2005', 'Kernel name': 'side-6973-eth1'}' - Unrecognised PCI address 'usb-0000:00:14.0-2'
interface-rename
-command errorA working solution was proposed by Alan Lantz as set out in [3] and the following write-up.
Fixed eth1 Address For USB NICs
To secure a fixed address for the extra USB NIC on each host, we need to do the following:
- Stop dynamic naming of NICs at boot.
- Establish static network rules.
- Make XenServer forget already registered NIC names that do not correspond to the names we want for the NICs.
Stop Dynamic Naming
Add net.ifnames=0
to the boot. Edit grub in /boot/grub/grub.cfg with either your favourite CLI editor, or by hitting e for edit at bootup.
serial --unit=0 --speed=115200 terminal_input serial console terminal_output serial console set default=0 set timeout=5 net.ifnames=0 menuentry 'XenServer' { search --label --set root root-aptphh multiboot2 /boot/xen.gz dom0_mem=752M,max:752M watchdog ucode=scan dom0_max_vcpus=2 crashkernel=128M@256M console=vga vga=mode-0x0311 module2 /boot/vmlinuz-4.4-xen root=LABEL=root-aptphh ro nolvm hpet=disable xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles module2 /boot/initrd-4.4-xen.img } menuentry 'XenServer (Serial)' { search --label --set root root-aptphh multiboot2 /boot/xen.gz com1=115200,8n1 console=com1,vga dom0_mem=752M,max:752M watchdog ucode=scan dom0_max_vcpus=2 crashkernel=128M@256M module2 /boot/vmlinuz-4.4-xen root=LABEL=root-aptphh ro nolvm hpet=disable console=tty0 xencons=hvc console=hvc0 module2 /boot/initrd-4.4-xen.img }
Establish Static Network Rules
Any persistent rules in 70-persistent-net.rules will thus be overruled by xen-backend.rules, Instead, create a new rule called 71-persistent-net.rules.
$ sudo vi /etc/udev/rules.d/71-persistent-net.rules SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="D4:6E:0E:06:2A:09", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
We now need to reboot to apply the naming rule for eth1. Just enter reboot
in the CLI to reboot.
We also need to define a static IP-address for the NIC, in order for it to work with the pool, by using xe pif-reconfigure
:
xe pif-reconfigure-ip uuid=4df1ed96-e208-1182-44ee-3a9bea79c1d4 mode=static IP=10.0.0.54 netmask=255.255.255.0 gateway=10.0.0.1 DNS=8.8.8.8
xe pif-reconfigure
.xe pif-list
.
Forget Unwanted NIC Names
After completing the steps above, we got two named NICs on the same MAC, one "side-xxx-1" and one eth1:
[root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label uuid ( RO): 07935775-8be8-6ce7-9551-edddb81a9c06 device ( RO): side-8373-eth1 host-name-label ( RO): xenserver-sampleserver02 uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4 device ( RO): eth1 host-name-label ( RO): xenserver-sampleserver02 uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a device ( RO): eth0 host-name-label ( RO): xenserver-sampleserver02
We need to employ xe pif-forget
to get rid of the "side-NIC" name. By using xe pif-forget
, we make sure that the "side-NIC" will be consistently removed throughout reboots:
[root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label uuid ( RO): 07935775-8be8-6ce7-9551-edddb81a9c06 device ( RO): side-8373-eth1 host-name-label ( RO): xenserver-sampleserver02 uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4 device ( RO): eth1 host-name-label ( RO): xenserver-sampleserver02 uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a device ( RO): eth0 host-name-label ( RO): xenserver-sampleserver02 [root@xenserver-sampleserver02 ~]# xe pif-forget uuid=07935775-8be8-6ce7-9551-edddb81a9c06 [root@xenserver-sampleserver02 ~]# xe pif-list params=uuid,device,host-name-label uuid ( RO): 4df1ed96-e208-1182-44ee-3a9bea79c1d4 device ( RO): eth1 host-name-label ( RO): xenserver-sampleserver02 uuid ( RO): d84e26aa-e7af-f059-6692-a3ec344cde7a device ( RO): eth0 host-name-label ( RO): xenserver-sampleserver02
xe pif-forget
.Finally, let's check that the network setup is correct:
[root@xenserver-sampleserver02 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000 link/ether 2c:56:dc:ba:f3:e9 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN qlen 1000 link/ether d4:6e:0e:06:e9:36 brd ff:ff:ff:ff:ff:ff 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1 link/ether 46:89:64:8f:a7:a9 brd ff:ff:ff:ff:ff:ff 5: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1 link/ether 2c:56:dc:ba:f3:e9 brd ff:ff:ff:ff:ff:ff inet 10.0.0.53/24 brd 10.0.0.255 scope global xenbr0 valid_lft forever preferred_lft forever 6: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1 link/ether d4:6e:0e:06:e9:36 brd ff:ff:ff:ff:ff:ff inet 10.0.0.54/24 brd 10.0.0.255 scope global xenbr1 valid_lft forever preferred_lft forever
ip a
.You can also get an overview of the network setup in XenCenter or in the xsconsole via the CLI:

We got two working NICs, eth0 and eth1, both with a static network setup. All is well.
Conclusions
We can avoid dynamic addition of NIC-names for USB NICs by adding net.ifnames=0
to grub in /boot/grub/grub.cfg, setting a static network configuration with xe pif-reconfigure
, and forgetting unwanted NIC-names with xe pif-forget
.
Changes to grub.cfg may be overwritten by XenServer updates and patches. You have to check if a patch or an update has affected the setup.
Figures
- Figure 1 - XenServer multipath setup with one NIC via USB
- Figure 2 - XenServer xsconsole network overview
Listings
- Listing 1 -
interface-rename
-command error - Listing 2 - net.ifnames=0 in grub.cfg>
- Listing 3 - Edit persisten network rules
- Listing 4 - Setting a static network confguration with
xe pif-reconfigure
. - Listing 5 - "side-xxx-1" NIC-name
- Listing 6 - Forgetting a NIC-name with
xe pif-forget
. - Listing 7 - Listing the network status with
ip a
.
Changelog
- 2017.11.03 - Added that a reboot is required after editing the renaming rules for eth1.
- 2017.11.21 - Added warning at the beginning of the article.
References
- [1]How to Change Order of NICs in XenServer, Downloaded October 24, 2017 - https://support.citrix.com/article/CTX135809
- [2]USB NIC with XenServer 7?, Downloaded October 24, 2017 - https://discussions.citrix.com/topic/379716-usb-nic-with-xenserver-7/
- [3]USB NICs In XS Pool Feasible?Published October 19, 2017 - https://discussions.citrix.com/topic/390537-usb-nics-in-xs-pool-feasible/