Proxmox VE 4.x OpenVZ to LXC Migration

September 4th, 2016 by Philip Iezzi 6 min read
cover image

At Onlime Webhosting we chose ProxmoxVE as our favorite virtualization platform and are running a bunch of OpenVZ containers for many years now, with almost zero issues. We very much welcome the small overhead and simplicity of container based virtualization and wouldn’t want to move to anything else. ProxmoxVE added ZFS support by integrating ZFSonLinux back in Feb 2015 with the great ProxmoxVE 3.4 release – which actually would have deserved to bump its major version because of this killer feature.

OpenVZ on PVE

Previously we have been running our OpenVZ containers on plain Debian Linux boxes. But with Debian Wheezy (7.0) the Debian community decided to no longer include a kernel which has been patched with the OpenVZ extensions. Switching to ProxmoxVE at that time was a piece of cake as ProxmoxVE was just a plain Debian Linux with a RHEL kernel, OpenVZ and KVM support, and some nice web interface to manage the containers and VMs. Actually, we never really needed the ProxmoxVE web interface (GUIs always suck!), but that one is quite lightweight and very well integrated. If you don’t like it, use the provided CLI tools – ProxmoxVE does not force you to use the web interface at all.

In 2014 I did a review for Proxmox High Availability (by PACKT publishing). But at Onlime, we never employed any cluster technologies to date. I was always looking for simple solutions without unnecessary complexity. ProxmoxVE with OpenVZ on ZFS pools was the perfect match. We are replicating our containers from one host node to another every 10mins with simple and super fast zfs send|receive via a modified version of ZREP.

In ProxmoxVE 4, OpenVZ support was removed in favor of LXC. Both, KVM and LXC are built into any newer Linux kernel, so in the long term (and due to the nature of OpenVZ as a huge kernel patch which is hard to maintain) it was clear we had to give up OpenVZ. Even though the OpenVZ developers are currently focusing on merging the OpenVZ and Virtuozzo source codebase and have just released OpenVZ 7.0 I don’t give OpenVZ a long future any more. In comparison to LXC, the tools for OpenVZ are still much more mature and powerful, but lately LXC made a lot of progress and having it perfectly integrated into ProxmoxVE is a great step forward. So, we have decided to migrate to LXC.

(Sorry about the long preface. You probably don’t care about history reading this blog post. I am getting to the point now…)

The clumsy (official) migration path

What I cannot believe and what prevented me to look into LXC on ProxmoxVE up to now is the clumsy migration path that ProxmoxVE suggests on ProxmoxVE Wiki: Convert OpenVZ to LXC. Basically they suggest to dump and restore a full container as follows:

pve1$ vzctl stop 100 && vzdump 100 -storage local
pve1$ scp /var/lib/vz/dump/vzdump-openvz-100.tar pve2:/var/lib/vz/dump/
pve2$ pct restore 100 /var/lib/vz/dump/vzdump-openvz-100.tar

Srsly? Do you guys only have containers with sizes below 1GB and don’t care about downtime at all? Real world looks different and I would not even go this migration path with a 5GB container. I don’t even want to think about those containers with ~ 1TB of data.

If you are running OpenVZ containers on ProxmoxVE, sure you have some easy solution to migrate your containers from one host node to another. As I said, we are using ZREP for container replication and have written a wrapper script that migrates a container to its failover host node. That’s what I suggest and please forget about live migration – it never really was working in OpenVZ (well, it was, but I am talking about a really stable solution that never fails) and LXC lacks any kind of hot/live migration. Doing a ZREP presync, stopping the container on the source host node, doing the main ZREP sync, finally starting the container on the destination host node is just a matter of seconds, usually causes a downtime of 10s up to 30s max (on a larger container).

Here’s the migration path I suggest for any ProxmoxVE setup with containers running on ZFS pools:

  1. move all OpenVZ containers to the primary host node
  2. upgrade the secondary host node to ProxmoxVE 4.x
  3. prepare the new LXC container on the secondary host node, don’t start it yet
  4. replace the newly created ZFS pool with the existing (which is still in sync with ZREP replication)
  5. do a vzctl stop $VEID && zrep failover rpool/ROOT/pve-$VEID to activate (rw) the ZFS pool on the secondary host node
  6. start the LXC container on the secondary host node
  7. once you have migrated all containers to LXC on the secondary host node, upgrade the primary host node to ProxmoxVE 4.x

On a new or upgraded ProxmoxVE host node, first configure ZFS storage:

pve2$ zfs create rpool/zfsdisks

# turn on posixacl by default for all containers (inherited)
pve2$ zfs set acltype=posixacl rpool/zfsdisks

pve2$ pvesm add zfspool zfsvols -pool rpool/zfsdisks -content images,rootdir -sparse

Check the storage configuration, /etc/pve/storage.cfg:

/etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    maxfiles 0
    content rootdir,iso,images,vztmpl

zfspool: zfsvols
    pool rpool/zfsdisks
    content images,rootdir
    sparse

Creating a new LXC container can be done via ProxmoxVE web interface or (my preference) via CLI. We are using the new zfsvols storage – pct takes care of creating the ZFS pool for you:

pve2$ pct create 100 local:vztmpl/debian-8.0-standard_8.4-1_amd64.tar.gz \
 -hostname demo.onlime.ch \
 -cpulimit 2 \
 -cpuunits 1024 \
 -memory 4096 \
 -swap 1024 \
 -ostype debian \
 -rootfs zfsvols:subvol-100-disk-1,acl=1 \
 -net0 name=eth0,bridge=vmbr0,gw=X.X.X.X,ip=X.X.X.100/25 \
 -storage zfsvols

You probably want to write a short helper script to simplify this step as most parameters will stay the same for all containers.

Do not try to change the subvol-VMID-disk-1 naming scheme of the LXC container, see Dietmar Maurer’s comment:

I guess we hardcoded that naming scheme a several places, so it would be hard to change that. We also use disk names to encode some informations, so we cannot allow arbitrary name changes.

We now have both ZFS pools on the secondary host node:

pve2$ zfs list
NAME  USED  AVAIL  REFER  MOUNTPOINT
rpool/ROOT/pve-100  560M  436G  529M  /var/lib/vz/private/184
rpool/zfsdisks/subvol-100-disk-1  560M  436G  529M  /rpool/zfsdisks/subvol-100-disk-1

Migration is now that simple, just replace the ZFS pool with your existing data (without dumping/backupping/restoring anything!):

pve2$ VMID=100
pve2$ zfs destroy -r rpool/zfsdisks/subvol-$VMID-disk-1
pve2$ zfs rename rpool/ROOT/pve-$VEID rpool/zfsdisks/subvol-$VMID-disk-1
pve2$ zfs set mountpoint=/rpool/zfsdisks/subvol-$VMID-disk-1 rpool/zfsdisks/subvol-$VMID-disk-1

After adjusting the zrep:dest-fs ZFS property on both source and destination host, you can stop the OpenVZ container, replicate it over, and start it as LXC container on the destination side:

pve1$ zfs set zrep:dest-fs=rpool/zfsdisks/subvol-100-disk-1 rpool/ROOT/pve-100
pve2$ zfs set zrep:dest-fs=rpool/zfsdisks/subvol-100-disk-1 rpool/zfsdisks/subvol-100-disk-1

pve1$ vzctl stop 100 && zrep failover rpool/ROOT/pve-100 && ssh pve2 pct start 100

That’s it. We didn’t have to copy a single bit of data and we just had the regular (very minimal) downtime of a container failover like we did it before in our OpenVZ-to-OpenVZ environment.