Setting up a development server with Xen 4.1.3

This is a detailed installation log for a development machine that I installed recently. I bought a second-hand Core2Duo PC with 4GB RAM and two SATA disks for this purpose. Additional 4GB RAM and two new disks (the old ones are too noisy) added 200% to the total cost :)


  1. Quick way of setting new virtual machines on demand
  2. Xen Paravirtualization is preferred because of a) better CPU performance; b) proven correct functioning of FreeSWITCH
  3. At least one guest VM needs access to the machine’s USB ports: for embedded Linux projects, I need a fast way to write an SD card

I’ve chosen to use the Debian Unstable as the Xen Dom0 controller, for the sake of testing the new things. As of this writing, Xen 4.1.3rc1 is available in Debian Sid.

It appeared that for a PV guest, there’s no stable way to have access to the host’s USB devices. The methods that one could try (I did) are described in Xen Wiki. So, if you only need storage and network, PV is fine, but if you need access to USB devices, HVM is the only stable choice.

Xen-4.1.3 comes with a choice between XM amd XL toolsets. I tried XL, but had to turn back to XM later. Problems with XL: 1) if you reboot the PV guest, it doesn’t come back online, and the VM needs to be created from dom0; 2) when a new PV VM starts, XL creates about 20MB of temporary data in /var/run/udev/data, and does not clean it up.

1. Set up the Dom0 system

Install the Debian OS:

  1. dd the installer image onto an USB flash drive
  2. Boot from the flash drive (Caveat: the flash drive is shown as /dev/sda, so the hard drives are /dev/sdb and /dev/sdc. When the installer sets up the Grub in MBR, it tries to write it to /dev/sda and fails. You need to pull out the USB flash drive and re-try installing GRUB).
  3. Choose the manual disk partitioning
  4. Delete all existing partitions from the hard drives
  5. Create a RAID1 device on top of the whole drives, no DOS partitioning necessary
  6. Create a Logical Group (LG0) on top of md0
  7. Create two Logical Volumes: LV0 (8GB), LV1 (1GB)
  8. Indicate that LV0 should be used as ext4 root partition, and LV1 should be used as swap
  9. The installer will ask which system components should be installed. Deselect all except “SSH server”
  10. Upon installing GRUB in the MBR, it may fail — remove the flash drive and repeat

After rebooting, you would have a basic Debian system on top of a RAID1 array which is most probably still resyncing (cat /proc/mdstat). It’s not necessary to wait untill resync finishes.

As indicated in Xen installation guide,

apt-get install xen-linux-system
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

Set up the bridge in /etc/network/interfaces  as described in Xen Wiki:

auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet static
        bridge_ports eth0

Set the memory limits for dom0 machine. The memory issues and the “max” parameter are explained in this article.

cat >>/etc/default/grub <<EOT

Update GRUB on both disks:

grub-install /dev/sda
grub-install /dev/sdb

When a guest VM starts, Xen creates about 20MB of temporary data in /run/udev/data/ (it’s cleaned up when the VM is running), and by default /run partition has the limit of 10% of the available RAM. These limits need to be adjusted in /etc/default/tmpfs:


Edit /etc/default/xendomains:


Create directories where VM configurations are to be stored and edited. It also makes sense to keep the “manual” directory under Git control:

# /etc/default/xendomains specifies /etc/xen/auto for automatically started guests. 
# We place symlinks from manual into auto.
mkdir /etc/xen/auto
# here we place configs for manually started guests
mkdir /etc/xen/manual
# set up Git
apt-get install git
cd /etc/xen/manual
git init

Reboot the machine.

After coming up, “xm list” should show “Domain-0″ machine with 255MB memory, and “df -h” should show 89M available for /run, and root filesystem on /dev/mapper/VG0-LV0 (around 8GB in size).

2. Install PV Ubuntu guest

Further on, “vlab1″ is the guest machine name.

lvcreate -L 40G -n vlab1 VG0
mkdir -p /var/lib/xen/images/ubuntu-netboot
cd /var/lib/xen/images/ubuntu-netboot

# this VM config is only used to launch Ubuntu installer
cat >/etc/xen/manual/vlab1-install.cfg <<'EOT'
name = "vlab1"
memory = 2048
vcpus = 2
disk = ['phy:/dev/VG0/vlab1,xvda,w']
vif = [' ']
kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"

# run the installer
xm create /etc/xen/manual/vlab1-install.cfg -c
# in Ubuntu partition manager, choose "Guided - use entire disk and set up LVM"
# When the installer finishes and reboots into the installer prompt again, do:
xm destroy vlab1
# Prepare the persistent config with automatic launching. 
# Make sure the bootloader points to an existing file.
cat >/etc/xen/manual/vlab1.cfg <<'EOT'
name = "vlab1"
memory = 2048
vcpus = 2
bootloader = "/usr/lib/xen-4.1/bin/pygrub"
disk = ['phy:/dev/VG0/vlab1,xvda,w']
vif = [' ']
ln -s /etc/xen/manual/vlab1.cfg /etc/xen/auto/
xm create /etc/xen/manual/vlab1.cfg -c

3. Install HVM Ubuntu guest

In this section, “vlab3″ is the VM name. If you don’t have X11 forwarding, you can launch the VNC viewer from your desktop (the VNC binding IP address  needs to be updated in /etc/xen/xend-config.sxp).

apt-get install vncviewer
lvcreate -L 40G -n vlab3 VG0
cd /var/lib/xen/images/ubuntu-netboot

# Ubuntu 12.04 already recognizes itself as a XEN HVM guest, and it contains PVHVM drivers. 
# So, the task is quite easy.

# Installer config:
cat >/etc/xen/manual/vlab3-install.cfg <<'EOT'
name = "vlab3"
memory = 2048
vcpus = 2
kernel ='/usr/lib/xen-4.1/boot/hvmloader'
device_model = '/usr/lib/xen-4.1/bin/qemu-dm'
disk = ['phy:/dev/VG0/vlab3,xvda,w', 'file:/var/lib/xen/images/ubuntu-netboot/mini.iso,xvdc:cdrom,r']
vif = ['bridge=xenbr0, mac=00:16:3f:03:05:14']

xm create /etc/xen/manual/vlab3-install.cfg
xm vncviewer vlab3&
# follow the standard installation process. The mini.iso will install base Ubuntu (even without ssh daemon).
# after it finishes, run "sudo apt-get install openssh-server" in the guest.

# set up production config with automatic start
cat >/etc/xen/manual/vlab3.cfg <<'EOT'
name = "vlab3"
memory = 2048
vcpus = 2
kernel ='/usr/lib/xen-4.1/boot/hvmloader'
device_model = '/usr/lib/xen-4.1/bin/qemu-dm'
disk = ['phy:/dev/VG0/vlab3,xvda,w']
vif = ['bridge=xenbr0, mac=00:16:3f:03:05:14']

ln -s /etc/xen/manual/vlab3.cfg /etc/xen/auto/
xm shutdown vlab3
xm create /etc/xen/manual/vlab3.cfg

At this point, you have a virtual machine with USB pass-through. With the following commands, the USB card reader is added to the VM:

xm usb-add vlab3 host:048d:1336
About these ads

, ,

  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: