I’ve been using ESXi as my virtualization platform in my homelab for years, 2012 maybe? It’s been a solid, but limited, platform. With Broadcom’s recent purchase, I figured it was time to start exploring more open source solutions for virtualization. I feel it’s only a matter of time before the free tier of ESXi is further limited.

I tried XCP-ng a few years ago, but I wasn’t able to get a virtual machine’s network interface into promiscuous mode for whatever reason. So this time, I think I’m going to set up Proxmox. I have 2 enterprise servers, so I want to set up a proxmox “cluster” (a true cluster needs 3 servers, so this will just be 2 servers in a “datacenter” that is my office). I want to be able to experiment with live migrating VMs from 1 host to another.

Installing Proxmox

This was pretty straight-forward. Follow the documentation.

Once Proxmox was set up, I configured an NFS share on my TrueNAS server. This will be the shared storage between the 2 proxmox hosts.

Multiple VLANs over a Single Interface

My network is segregated into multiple VLANs: Home, IoT, 2 work VLANs and a guest VLAN. I also have a physically isolated DMZ network. To get all of this into Proxmox without using 6 Network interfaces, I found some steps on how to enable multiple VLANs over a single interface.

Edit your /etc/network/interfaces file

On my server, I have vmbr0 set to my main network interface. This is the interface I want the multiple VLANs to come in on.

For each VLAN, use the template below. Insert it above the vmbr0 definition in the file.

auto vmbr0.vlan-id
iface vmbr0.vlan-id inet static
    address xxx.xxx.xxx.xxxx/24 # IP for the interace on the VLAN subnet
    gateway xxx.xxx.xxx.xxx # IP Address for the gateway on the VLAN

Finally, in the vmbr0 definition, add the following lines to enable the VLANs. You can specify specific VLANs here, but I enabled all of them for simplicity. However, if you only wanted to allow 2 VLANs (say 10 and 20) you would use bridge-vids 10 20

bridge-vlan-aware yes
bridge-vids 1-4092 # the VLAN IDs that will be allowed on this interface.

Once this is done, I think rebooting the system would be best to ensure the configuration is applied properly.

Migrating VMs from ESXi to Proxmox

This is going to be the time consuming process:

  • Uninstall VMware Tools. Most of my linux systems are a debian flavor so it’s a simple apt purge open-vm-tools
  • Shutdown the VM
  • Export the vm using the VMWare ovftool
  • Copy the resulting VMDK files to Proxmox
  • Create a VM in Proxmox to match the original VM settings
  • Detach, and remove, the hard disk from the blank Proxmox VM
  • From the Proxmox CLI, run qm importdisk vm-id path-to-disk proxmox-storage -format qcow2
  • Repeat the prior step for all disks, if more than 1
  • In the Proxmox GUI, double click on the “unused disk”, select Add
  • Repeat the prior step for all disks, if more than 1
  • Enable the new drives in Options > Boot Order

Start up the New VM

Once all of this is complete, you can power on the VM and make sure everything imported and starts properly.

I use netplan to control my interface configuration. Changing to Proxmox relabeled my interfaces, so I had to go into my netplan configuration /etc/netplan/01-ifcfg.yaml to update the interface name and then run sudo netplan apply to apply the new configuration. I have a blog post about netplan if you want more of a step-by-step here.

Repeat for each VM. I have 15 or so I think I use on a regular basis, so I’ll be on this step for a while.

Thoughts so far

The only feature ESXi has that I haven’t seen a replacement for, is using VMWare Player as a remote console. I really enjoyed that and used it all the time. I’ve recently started using Guacamole as a centralized remote access server so that’s helping.