A guide on how to install and operate a two-host XenServer pool with manual fail-over. This is the first part of a series of articles on a small-footprint XenServer setup with no single points of failure.
When would you consider a two-host XenServer pool? In most cases, a high-access (HA) setup with 3 or more hosts would be preferable, because that gives you automatic fail-over. However, the two-host solution might be your choice if;
You need redundancy, but HA is overkill. If you have time to switch over manually, HA may be unnecessary.
Budget restrictions means a 3-host (or more) solution is undoable.
You simply can't find a third host (or more) that will work with the two hosts you already have. Sounds odd? Wait until you see the XenServer requirements for hardware in a pool...
A 2-host solution is a bit simpler to maintain than a HA solution.
“A resource pool comprises multiple XenServer host installations, bound together into a single managed entity which can host Virtual Machines. When combined with shared storage, a resource pool enables VMs to be started on any XenServer host which has sufficient memory and then dynamically moved between XenServer hosts while running with minimal downtime (XenMotion). If an individual XenServer host suffers a hardware failure, then the administrator can restart the failed VMs on another XenServer host in the same resource pool.” 
Note that only if high availability (HA) is enabled on the resource pool, will VMs be automatically moved if their host fails.
Manual fail-over requires a minimum of 2 XenServer hosts – only the master switched on – with a XenServer storage repository (SR) readily available:
SR (storage repository: A XenServer storage repository (SR) is a storage container on which virtual disks are stored. Both storage repositories and virtual disks are persistent, on-disk objects that exist independently of XenServer. SRs can be shared between servers in a resource pool and can exist on different types of physical storage devices, both internal and external, including local disk devices and shared network storage. 
SSH: Establish SSH with certificates to the SR host in the system. Increase security by disabling password login and base everything on client and server certificates. See  for details (Ubuntu server reference installation).
A properly configured firewall: The firewall protecting the subnet must be configured so that VPN and SSH are workable. Moreover, to the degree that firewalls are enabled on each hosts, these must also allow for necessary services, including SSH. For example, see  for details on the Ubuntu UFW Firewall.
A XenServer joining a pool;
is not a member of an existing resource pool,
has no shared storage configured,
has no running or suspended VMs,
has no active operations on the VMs in progress, such as one shutting down,
its management interface is not bonded,
its management IP address is static,
is running the same version of XenServer software, at the same patch level, as servers already in the pool, see Preparing VMs for Use in a Pool below for details, and
the clock of the host joining the pool is synchronized to the same time as the pool master.
Host hardware requirements: The CPUs on the server joining the pool are the same (in terms of vendor, model, and features) as the CPUs on servers already in the pool (with some variation possible, see ). The details on each CPU relevant for XenServer can be checked in the CLI of each XenServer.
It is possible to create pools with slightly different hosts. Whether a so-called heterogenous setup actually works has to be thoroughly tested. If you can, use the most homogenous setup possible.
Installation and Configuration
So how do we get started? Let's start by using this hardware setup:
It is possible to run XenServer on very limited hardware setups. Running a pool with mini-PCs like Intel NUCs or Asus VivoMinis is a real possibility, and not only for the lab, but for real-world applications. Note that if you chose to go with hardware setups that are not included in the XenServer list of approved hardware, those setups need to be tested.
Create XenServer Pool
Step 1 - Check Prerequisites
In order to access the XenServer hosts that we are going to use to use for the pool, we need to
access each host through XenCenter,
access each host through SSH using the console,
use both XenCenter and SSH access, or
work with xsconsole, the local terminal directly on the server.
In the following, I will show both XenCenter and SSH access. I will mention xsconsole whenever it is the fastest route to solve something.
Note that there are some things that are easier and faster to do via the command line interface (CLI), while others are better done via XenCenter. The CLI is especially useful if you are dealing with lots of hosts and / or lots of VMs.
Connecting to the XenServer hosts via SSH: To get started, open the terminal and log into the first XenServer host (in this case samplehost01):
Open a new window in the terminal and log into it (in this case samplehost02):
In Xencenter:Start XenCenter and connect to your hosts using Server - Add. After filling out server-IP and credentials, the hosts should become visible in the left frame.
A. Is the hardware of the hosts similar?
We can use xe host-cpu-info to figure out if the CPUs are similar. For samplehost01 we get:
For samplehost02 we get:
Not only are these CPUs similar, they are identical! Can we proceed? Not yet; we also need to check networking, available memory and storage. Have a look at networking with lspci | egrep -i --color 'network|ethernet',available memory and memory type with free -m and dmidecode -t 17, and storage hardware with lsblk -o KNAME,TYPE,SIZE,MODEL. On our sample-hosts we get:
If we use the same commands on samplehost02, we get exactly the same results. What luck! We are not only dealing with siblings here; we are dealing with identical twins! Let's see whether our luck holds. On to the next hurdle.
XenCenter: Look at CPU and memory under the General-tab, disk(s) under the Storage-tab, and network card(s) under the NICs-tab, and compare.
B. Are the hosts members of an existing resource pool?
Pools and members of pools can be clearly seen in XenCenter. If you connect to a server via it's IP-address, XenCenter will show any eventual pool of which the server is a member. Alternatively, you can use xe pool-list in the CLI to get a list of pools, if any. As we can see from the figure above shwoing the CLI, there are no pools present. We can also check by using xe pool-list in the CLI:
Looks like both hosts are members of an existing pool! But hold on; when a host is not a member of any resource pool, it is a master of its own pool. This pool has no name (nothing in the "name-label ( RW)"-entry) and the UUID of the master is the same as the host. Compare the UUIDs below with the UUID of the masters above:
In other words, the hosts are not members of any existing resource pool, and this can also be seen in the XenCenter, where no pool is shown. Moreover, if a host is a member of a pool, and there are more members of a pool, all members are shwon when using xe host-list. We can proceed to the next hurdle.
In XenCenter: If you connect to a host, and that host is a member o a pool, the hosts appear under the pool. Example:
C. Do any of the hosts have a shared storage?
Let's have a look at the storage repositories on each host, using the xe sr-list command:
Lots of information, but it is all about local storage. No shared storage in sight. Everything in order. Next!
XenCenter: Eventual shared storage can also be seen in XenCenter, under the storage-tab:
D. Are there any running or suspended VMs on any of the hosts?
To figure out which VMs are on the hosts, and which state each VM is in, we use xe vm-list. Entering it in the samplehost02 CLI, we get:
This host has only the control domain running. There are no VMs. However, if we run xe vm-list on samplehost01, we get a different result:
This host has the control domain running, In addition, there are two VMs, but they are neither running nor suspended, so we are OK. Next!
XenCenter: You can clearly see running or suspended VMs in the top left panel (expand each host to see all VMs). In this case, the VMs are clearly halted.
E. Are there active operations on the VMs in progress on any of the hosts?
On one host we have no VMs, and on the other all VMs are halted, so nothing could be going on, right? Wrong. One or several of the VMs might be in the process of being moved, copied, or exported, or a snapshot of the VM may be in progress. To check if any process is active, use the list_domains-command:
On samplehost02, only the control domain is running (The uuid listed is the control domain as seen in D above.). On samplehost01, something is going on, and we have to wait until the process has terminated - or stop the process - before we move on. If you have trouble terminating a process, see Troubleshooting below.
As an alternative to list_domains, we can use xentop. The advantage of this command is that it gives us a console window that is continuously updated with the current processes on the host, in this case dom0 and two VMs:
XenCenter: Active operations can be seen on the bottom line by choosing a VM. If anything appears, there is an active process in progress.
F. Is the hosts' management interfaces bonded?
This is an easy one. Just check if there are bonded interfaces with xe bond-list:
Nothing in both cases. If something shows up here, it can be removed with xe bond-destroy.
G. Is the management IP address of each host static?
The fastest way of checking whether the IP address is static is to use the XenServer CLI from the SSH console. Just type xsconsole and the xsconsole shown in the figure appears. Use the down-arrow to move to Network and Management Interface. We can see that in this case, the IP address is static. The same is the case for samplehost02. Next!
In XenCenter: Click on the networking tab for details. Under IP Setup, we can see that samplehost01 has a static ip setup.
H. Are the hosts running the same version of XenServer software, at the same patch level?
We can get a lot of data on each host by using xe host-param-list, but it needs the host's UUID. Let's get that firrst:
Then we use the UUID as aparameter in xe host-param-list:
That's a lot of information in one place. However, we are just looking for the XenServer version and the patch level. That information is available under software-version (MRO) and updates:
So what does the samplehost2 information look like? Using the same command in the samplehost2 CLI, we get:
They are identical. Next!
XenCenter: The version and patch level can be checked underthe General-tab:
I. Are the clocks of the hosts synchronized to the same time?
The fastest way of checking whether the time is synchronized on the hosts is to use the XenServer CLI from the SSH console and check whether NTP has been enabled. Just type xsconsole and the xsconsole shown in the figure appears. Use the down-arrow to move to Network and Management Interface. We can see that in this case, NTP has been activated. The same is the case for samplehost02. Moreover, we can simply check the time by looking at the top of the xsconsole. Both xsconsoles show the same time.
Alternatively, we could check that the hosts actually have the same time and time-zone - and that NTP is running - via the CLI:
The NTP is active and the clocks on both hosts are the same (it takes a few fractions of a second to type date in the subsequent CLI, hence the small difference). We have reached the end of the requirements and are ready to deploy.
Step 2 - Creating the Initial Pool
A minimum of 2 hosts are needed for manual fail-over. Up to 16 hosts are supported per resource pool.
We get going by gathering some basic information; hostname, host UUID, and the UUID of the initial pool on the host.
As we saw in Step 1 - B, the host is not part of any pool if the name-label of the initial pool is empty:
We can go ahead and create a pool with the host samplehost01 as the poomaster simply by giving the initial pool a name-label:
Let's just make sure that it worked:
It did. We can now go ahead and add the other host - samplehost02 - to the pool:
That worked. There are now two hosts visible from each host.
In XenCenter:Creating a new pool is simple. From the XenCenter, simply click on New Pool and follow the instructions:
Step 3 - Add a Storage Repository
In this example we will be using an NFS share. Create an NFS file share on a bare-metal Linux server, see  for details. Then, create an NFS VDH Storage on the NFS share as set out in  and , page 22. In brief, adding NFS shared storage to a resource pool using the CLI is done as follows:
Open a CLI on any XenServer host in the pool.
Create the storage repository on by issuing the command:
Find the UUID of the pool by the command $ xe pool-list
Set the shared storage as the pool-wide default with the command xe pool-param-set uuid= default-SR=
Our NFS-share is on 10.0.0.63 with the path /opt/share:
Check that the command was successful:
The sampleSR is visible and shared. Since the shared storage has been set as the pool-wide default, all future VMs will have their disks created on shared storage by default. See , Chapter 5, Storage, for information about creating other types of shared storage.
The device-config:server refers to the hostname of the NFS server and device-config:serverpath refers to the path on the NFS server. Since shared is set to true, the shared storage will be automatically connected to every XenServer host in the pool and any XenServer hosts that subsequently join will also be connected to the storage. The Universally Unique Identifier (UUID) of the created storage repository will be printed on the screen.
XenCenter: You add a SR simply by right-clicking the pool, choosing New SR, and then just following the dialog.
Step 4 - Adding Hosts to the Pool
You add a host to the pool by using the xe pool-join-command. Only servers that are not a part of another pool - and that comply with all the other prerequisites above - can be added. Before adding the host to a pool, the world looks like this seen from the host to be added:
Only the host we are on is visible, and only the non-active, nameless pool that is part of any XenServer host is shown. Now let's add this host to a pool and see what happens:
Let's give it a few seconds. 10 o be precise. After that, list pools and hosts:
We are good. The pool that is visible now is the one we have joined. The empty, non-active pool isn't listed anymore. Moreover, both samplehost01 and samplehost02 are now visible in the host-list.
If you added a host by mistake, you can eject that host:
The part about attempting to restart as a master of a new pool just means that the host will restart as a master of a nameless pool, that is, the default non-active pool that is a part of all XenServer hosts.
XenCenter: You add a host to the pool by right-clicking on the pool and choosing Add server.
Step 5 - System Restart and Check
To restart the pool, for each host in the pool do the following:
Check: List the pool and hosts on each host. The pool and the hosts should be visible as before the restart.
XenCenter check: Right-click on the XenServer instances and click reboot.
What you have seen is pretty much what is required to set up a pool; installation of XenServer on two hosts, and a shared storage repository. The next step is to install and use VMs. That will be the next article in this blog.
Note: We have not discussed the nitty-gritty of network setup for XenServer pools. That will be the theme of a separate article. The setup above will getyou going, but it is prone to network congestion.
Wishes? Opinions? Feedback as such? Please get in touch. If you want a particular theme discussed, let us know. The same goes if you want to see this article in another language. Thanks.
The problem with a pool with a shared storage is that some unwanted processes may be kept alive. For instance, if you start a VM export and then lose connectivity to the export storage, the process may hang. The steps to remedy the situation are as follows:
1. Find the UUID of the hung VM: # xe vm-list
2. Find the Domain ID of the hung VM.# list_domains
3. Match the UUID with the ID number:
4. Run destroy_domain on the Domain ID: # /opt/xensource/debug/destroy_domain -domid 62
5. The VM will still show itself as running, so now, we need to reboot it: # xe vm-reboot name-label='name of the VM' –force
The VM is now rebooted and you can start it again.
The Host Name Is Not Resolved When Attemtping to Join a Pool
A quick-fix for this problem is to use the IP address instead of the host name. In addition, you should have a look at your nameserver setup and check that hostnames really resolve on all hosts.
2017-10-02: Added the xentop-command to Step 1 - E.
2017-10-02: Added Listings-table and Figures-table.