KVM stands for Kernel-based Virtual Machine and is my preferred way of virtualizing a Linux system, in combination with libvirt. This article documents a few hints on what my current best practice setup is.

For the host and most VMs, I use Debian wheezy (which is not yet released as of writing this article). I want a recent kernel and, most importantly, systemd.

Networking

An easy setup is to just use a bridge which contains the real ethernet device (eth0). However, I have noticed a few problems with IPv6 routes on bridge devices which turned out to be kernel bugs (1, 2, possibly I forgot some). The symptom is similar to this post by Marc, essentially leading to VMs which don’t come up properly after rebooting the host or even rebooting just the VM.

Therefore, I use tap devices instead. They are a bit complicated to setup. See the libvirt documentation on generic ethernet connections and this libvirt wiki page, which explains how you can lower the host protection to be able to use these interfaces. Here is the XML part of the VM definition which I use:

<interface type='ethernet'>
  <mac address='52:54:00:fa:f1:f1'/>
  <script path='/etc/libvirt/qemu/networks/ifup-infra.sh'/>
  <target dev='tap.infra'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>

The script /etc/libvirt/qemu/networks/ifup-infra.sh contains:

#!/bin/sh
#
# Network configuration for kvm domain "infra"
#
# Generate a random (but fix) MAC address:
#     echo -n 'f6:'; openssl rand -hex 5 | sed 's/\(..\)/\1:/g; s/.$//'
# F6 is a prefix which is considered Locally Administered because the second
# lowest bit of the first byte is set to 1.

set -e

/sbin/ip link set $1 address f6:63:cb:bb:59:e1
/sbin/ip link set $1 up
/sbin/ip -4 route add 79.140.39.194/32 dev $1
/sbin/ip -4 route add 79.140.39.198/32 via 79.140.39.194 dev $1
/sbin/ip -6 route add 2001:4d88:100e:1::/64 dev $1
# RZL VPN
/sbin/ip -6 route add 2001:4d88:100e:ccc0::/64 via 2001:4d88:100e:1::2 dev $1
/sbin/ip -6 route add 2001:4d88:100e:ccc::/64 via 2001:4d88:100e:1::2 dev $1

The script sets a static, locally administered MAC address on the tap interface on the host. This is necessary so that the VM can use the same link-local address as default route.

IP Addresses

On the host, I have a /27 IPv4 network and a /48 IPv6 network. Previously, I used an IP address configuration such as 79.140.39.194/27 inside the VM. However, that only works when you have no special routes such as an IP address you route into a VPN (or you add routes within each VM which is error-prone and tiresome).

Therefore, I configure IPs in the VM with a /32 netmask, e.g. 79.140.39.194/32. The default IPv4 route within each VM is 192.168.23.23, so that I don’t lose one of my precious public IPs for that purpose:

host # ip -4 address add 192.168.23.23/32 dev lo
vm # ip -4 route add 192.168.23.23 dev eth0
vm # ip -4 route add default via 192.168.23.23

For IPv6, we use a link-local address, which depends on the MAC address we set earlier in ifup-infra.sh:

vm # ip -6 route add default via fe80::f463:cbff:febb:59e1 dev eth0

For the reference, here is the entire /etc/network/interfaces of a VM:

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
    address 79.140.39.194
    netmask 255.255.255.255
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 80.244.244.244
    dns-search in.zekjur.net
    post-up ip -4 route add 192.168.23.23 dev eth0
    post-up ip -4 route add default via 192.168.23.23
    post-up ip -4 address add 79.140.39.197/32 dev eth0
    post-up ip -6 address add 2001:4d88:100e:1::2/64 dev eth0
    post-up ip -6 address add 2001:4d88:100e:1::3/64 dev eth0
    post-up ip -6 route add default via fe80::f463:cbff:febb:59e1 dev eth0
    # iptables
    post-up iptables-restore < /etc/network/iptables
    post-up ip6tables-restore < /etc/network/ip6tables

Serial console

It is benefical to be able to use the virsh console so that you don’t have to use VNC to access your virtual machine’s text consoles. For VNC, you probably need to create an SSH-tunnel first and then you might have issues with your keyboard layout. Also, the virsh console is much faster since it’s text-only. Either way (virsh console or VNC) can be used to recover your VM in situations where you e.g. messed up /etc/network/interfaces and the VM is not reachable over the network anymore.

Luckily, with systemd inside the VM, the only thing you have to do is modify /etc/default/grub like this:

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0 init=/bin/systemd"

The order is important here! See also Lennart’s blog post about serial consoles with systemd.

In case you are not yet using systemd, you have to uncomment the getty entry for ttyS0 in /etc/inittab to get a login prompt.

After configuring your VM accordingly, you can use virsh console infra to access the VM called "infra" on serial console level.

Caching / Performance

Depending on your use-case, you might want to change the cache settings which can bring dramatic disk bandwidth improvements in the virtual machine at the expense of data security.

Also see the section "Performance Tuning" of Michael David’s post about his KVM setup for further tips on performance.

Snapshots (backup)

I previously wrote about snapshots to backup virtual machines and still use that approach. Unfortunately, there is a bug which prevents lvremove from working properly.

I updated my script to include a workaround which works in the vast majority of cases (I only once had to clean up block devices manually in a high-load scenario since deploying the workaround).

Find it at the xen-lvm-snapshot git repository.