About two years ago, I rented a dedicated server to host a few virtual machines. It’s been running fine ever since, but at some point, faster and more energy-efficient hardware is available for the same price. I decided to rent a new server mostly for the additional RAM, which is very valuable when hosting virtual machines — other resources such as CPU, disk and network are usually not a bottleneck.

This article documents the different steps I used to migrate my (KVM and libvirt) virtual machines from my old server (called OLD with IP address 192.168.2.2 from now on) to my new server (called NEW with IP address 192.168.9.9 from now on).

Note that I did not do a live migration because I don’t use network storage which is accessible from both machines. I needed to transfer the data from one machine to the other.

The plan

1. Transfer the virtual machine’s data to NEW.
2. Stop the virtual machine on OLD, boot it up on NEW.
3. Redirect traffic from OLD to NEW (the IPs are transferred to the new server later on).

The obvious problem is that the virtual machine is still running between step 1 and 2 and therefore it might modify the data. Normally, to avoid inconsistencies, you use an LVM snapshot, but that doesn’t help you with additional, consistent data which the virtual machine may write. The solution is described below, but let’s start with the tunnel setup first.

Traffic redirection: preparation

To be able to boot up the virtual machine on NEW and use it immediately (without contacting the hoster to change the routing table), we set up a tunnel from OLD to NEW. Beware: Depending on the routing setup at your hoster, there might be a reverse-path filter in our way. Such a filter blocks packets which come from the wrong path, in our case it might only allow packets originating from OLD, but not from NEW. Therefore, double-check first if you can use a tunnel like this at your hoster before you follow this article.

On OLD, add the following entries to /etc/network/interfaces:

auto legacy
iface legacy inet tunnel
	mode     gre
	local    192.168.2.2
	endpoint 192.168.9.9
	address  10.0.1.1
	netmask  255.255.255.0
	ttl      255

auto legacy6
iface legacy6 inet tunnel
	mode     sit
	local    192.168.2.2
	endpoint 192.168.9.9
	address  fd26:a975:9d12::1
	netmask  48

On NEW, add the same entries, but swap the "local" and "endpoint" fields and use a different IP address (e.g. 10.0.1.2 and fd26:a975:9d12::2). The IPv6 address is an automatically generated RFC4193 (Unique local address) address, generated with Holger Zuleger’s generate-rfc4193-addr.sh.

Afterwards, bring up the networks with ifup legacy and ifup legacy6 on OLD and NEW and check that they can ping each other.

Transferring the data

We will transfer the data in two steps: First, we just copy the whole block device over, then we shut down the virtual machine and copy only the differences. Copying the differences is usually done in a few seconds, while a block device transfer of a 10 GiB device takes about 20 minutes, so this technique reduces the total down time for the virtual machine to a minimum. If you tend to chose too big disk sizes for your VMs, waiting for a long time to transfer them might be a lesson to choose smaller sizes in the future… :-)

First, create the new logical volume and take a snapshot of the old one (the snapshot size determines how much data the virtual machine can write as long as it’s still running):

NEW # lvcreate -L 20G -n domu-web newhost
OLD # lvcreate --snapshot -L10G -n web-lvmsync oldhost/domu-web

Then, transfer the data, compressed, with a progress bar. I just love the power of pipes when seeing such a command :-).

OLD # dd if=/dev/oldhost/domu-web bs=1M | pv -ptrb -s 20G | \
      gzip -3c | ssh new '(gunzip -c - | dd of=/dev/newhost/domu-web)'

Transferring the differences

Afterwards, shut down the VM, use lvmsync to transfer the differences and boot the VM on NEW:

web # shutdown -h now
OLD # lvmsync /dev/oldhost/web-lvmsync new:/dev/newhost/domu-weg
NEW # virsh create /etc/libvirt/qemu/web.xml

You can copy the file web.xml from OLD and modify it, or, if you have many changes in your setup, start with a fresh one (pay attention to keeping the same MAC address).

Redirecting the traffic

The final step is easy: just add the appropriate route(s) on OLD (don’t forget to make them persistent in /etc/network/interfaces):

OLD # ip -4 route add 79.140.39.195/32 via 10.0.1.1 dev legacy
OLD # ip -6 route add 2001:4d88:100e:2::/64 via fd26:a975:9d12::1 dev legacy6

And we’re done! The total downtime of the VM is a few minutes — the time it needs to shut down, transfer differences and boot up again. Booting might take longer than normal if a file system check is necessary, thus it’s not done in a few seconds probably.