This is a detailed installation log for a development machine that I installed recently. I bought a second-hand Core2Duo PC with 4GB RAM and two SATA disks for this purpose. Additional 4GB RAM and two new disks (the old ones are too noisy) added 200% to the total cost
- Quick way of setting new virtual machines on demand
- Xen Paravirtualization is preferred because of a) better CPU performance; b) proven correct functioning of FreeSWITCH
- At least one guest VM needs access to the machine’s USB ports: for embedded Linux projects, I need a fast way to write an SD card
I’ve chosen to use the Debian Unstable as the Xen Dom0 controller, for the sake of testing the new things. As of this writing, Xen 4.1.3rc1 is available in Debian Sid.
It appeared that for a PV guest, there’s no stable way to have access to the host’s USB devices. The methods that one could try (I did) are described in Xen Wiki. So, if you only need storage and network, PV is fine, but if you need access to USB devices, HVM is the only stable choice.
Xen-4.1.3 comes with a choice between XM amd XL toolsets. I tried XL, but had to turn back to XM later. Problems with XL: 1) if you reboot the PV guest, it doesn’t come back online, and the VM needs to be created from dom0; 2) when a new PV VM starts, XL creates about 20MB of temporary data in /var/run/udev/data, and does not clean it up.
1. Set up the Dom0 system
Install the Debian OS:
- dd the installer image onto an USB flash drive
- Boot from the flash drive (Caveat: the flash drive is shown as /dev/sda, so the hard drives are /dev/sdb and /dev/sdc. When the installer sets up the Grub in MBR, it tries to write it to /dev/sda and fails. You need to pull out the USB flash drive and re-try installing GRUB).
- Choose the manual disk partitioning
- Delete all existing partitions from the hard drives
- Create a RAID1 device on top of the whole drives, no DOS partitioning necessary
- Create a Logical Group (LG0) on top of md0
- Create two Logical Volumes: LV0 (8GB), LV1 (1GB)
- Indicate that LV0 should be used as ext4 root partition, and LV1 should be used as swap
- The installer will ask which system components should be installed. Deselect all except “SSH server”
- Upon installing GRUB in the MBR, it may fail — remove the flash drive and repeat
After rebooting, you would have a basic Debian system on top of a RAID1 array which is most probably still resyncing (cat /proc/mdstat). It’s not necessary to wait untill resync finishes.
As indicated in Xen installation guide,
apt-get install xen-linux-system dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
Set up the bridge in /etc/network/interfaces as described in Xen Wiki:
auto lo iface lo inet loopback auto xenbr0 iface xenbr0 inet static bridge_ports eth0 address 192.168.1.40 netmask 255.255.255.0 network 192.168.1.0 gateway 192.168.1.1 dns-nameservers 220.127.116.11
Set the memory limits for dom0 machine. The memory issues and the “max” parameter are explained in this article.
cat >>/etc/default/grub <<EOT GRUB_CMDLINE_XEN="dom0_mem=256M,max:256M" EOT
Update GRUB on both disks:
update-grub grub-install /dev/sda grub-install /dev/sdb
When a guest VM starts, Xen creates about 20MB of temporary data in /run/udev/data/ (it’s cleaned up when the VM is running), and by default /run partition has the limit of 10% of the available RAM. These limits need to be adjusted in /etc/default/tmpfs:
Create directories where VM configurations are to be stored and edited. It also makes sense to keep the “manual” directory under Git control:
# /etc/default/xendomains specifies /etc/xen/auto for automatically started guests. # We place symlinks from manual into auto. mkdir /etc/xen/auto # here we place configs for manually started guests mkdir /etc/xen/manual # set up Git apt-get install git cd /etc/xen/manual git init
Reboot the machine.
After coming up, “xm list” should show “Domain-0” machine with 255MB memory, and “df -h” should show 89M available for /run, and root filesystem on /dev/mapper/VG0-LV0 (around 8GB in size).
2. Install PV Ubuntu guest
Further on, “vlab1” is the guest machine name.
lvcreate -L 40G -n vlab1 VG0 mkdir -p /var/lib/xen/images/ubuntu-netboot cd /var/lib/xen/images/ubuntu-netboot wget http://mirror.switch.ch/ftp/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/initrd.gz wget http://mirror.switch.ch/ftp/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/vmlinuz # this VM config is only used to launch Ubuntu installer cat >/etc/xen/manual/vlab1-install.cfg <<'EOT' name = "vlab1" memory = 2048 vcpus = 2 disk = ['phy:/dev/VG0/vlab1,xvda,w'] vif = [' '] kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz" ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz" extra = "debian-installer/exit/always_halt=true -- console=hvc0" EOT # run the installer xm create /etc/xen/manual/vlab1-install.cfg -c # in Ubuntu partition manager, choose "Guided - use entire disk and set up LVM" # When the installer finishes and reboots into the installer prompt again, do: xm destroy vlab1 # Prepare the persistent config with automatic launching. # Make sure the bootloader points to an existing file. cat >/etc/xen/manual/vlab1.cfg <<'EOT' name = "vlab1" memory = 2048 vcpus = 2 bootloader = "/usr/lib/xen-4.1/bin/pygrub" disk = ['phy:/dev/VG0/vlab1,xvda,w'] vif = [' '] EOT ln -s /etc/xen/manual/vlab1.cfg /etc/xen/auto/ xm create /etc/xen/manual/vlab1.cfg -c
3. Install HVM Ubuntu guest
In this section, “vlab3” is the VM name. If you don’t have X11 forwarding, you can launch the VNC viewer from your desktop (the VNC binding IP address needs to be updated in /etc/xen/xend-config.sxp).
apt-get install vncviewer lvcreate -L 40G -n vlab3 VG0 cd /var/lib/xen/images/ubuntu-netboot wget http://mirror.switch.ch/ftp/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/mini.iso # Ubuntu 12.04 already recognizes itself as a XEN HVM guest, and it contains PVHVM drivers. # So, the task is quite easy. # Installer config: cat >/etc/xen/manual/vlab3-install.cfg <<'EOT' name = "vlab3" memory = 2048 vcpus = 2 kernel ='/usr/lib/xen-4.1/boot/hvmloader' builder='hvm' device_model = '/usr/lib/xen-4.1/bin/qemu-dm' xen_platform_pci=1 disk = ['phy:/dev/VG0/vlab3,xvda,w', 'file:/var/lib/xen/images/ubuntu-netboot/mini.iso,xvdc:cdrom,r'] vif = ['bridge=xenbr0, mac=00:16:3f:03:05:14'] boot="dc" vnc=1 EOT xm create /etc/xen/manual/vlab3-install.cfg xm vncviewer vlab3& # follow the standard installation process. The mini.iso will install base Ubuntu (even without ssh daemon). # after it finishes, run "sudo apt-get install openssh-server" in the guest. # set up production config with automatic start cat >/etc/xen/manual/vlab3.cfg <<'EOT' name = "vlab3" memory = 2048 vcpus = 2 kernel ='/usr/lib/xen-4.1/boot/hvmloader' builder='hvm' device_model = '/usr/lib/xen-4.1/bin/qemu-dm' xen_platform_pci=1 disk = ['phy:/dev/VG0/vlab3,xvda,w'] vif = ['bridge=xenbr0, mac=00:16:3f:03:05:14'] usb=1 vnc=1 EOT ln -s /etc/xen/manual/vlab3.cfg /etc/xen/auto/ xm shutdown vlab3 xm create /etc/xen/manual/vlab3.cfg
At this point, you have a virtual machine with USB pass-through. With the following commands, the USB card reader is added to the VM:
lsusb xm usb-add vlab3 host:048d:1336