VMware ESXi to XCP-NG

Given the pending Broadcom acquisition of VMware, I have been giving some thought about an alternative for my homelab virtualisation solution. Given Broadcom’s previous approaches to acquisitions, given I am using the free version of ESXi I have concerns whether the free version will continue.

These concerns are more than just for the homelab user. Organisations who are VMware customers are also having concerns. As Ed Scannell of TechTarget SearchDataCenter noted in his article Broadcom’s acquire-and-axe history concerns VMware users (retrieved 2022 July 10): “Broadcom’s history: Acquire and axe”. This make perfect business sense: sell off or stop the unprofitable/low profitability products, concentrate on your top customers and maximize shareholder return. That is your fiduciary responsibility.

However, that doesn’t help the homelab user. (Let alone the SMB market…) How this all plays out, well time will tell.

I’ve been a big advocate of VMware starting back in the very early 2000’s. Back in the days of VMware GSX (basically what VMware Workstation is now) and ESX. Back then, looking at server utilisation where everything was physical showed a massive underutilization. “Old fogies” like me will remember your NetServers and Proliants with average CPU utilizations around 25% (or 5-10%…).

Basically, I have been using VMware ESX and later ESXi both professionally and personally for about 20 years. (Wow! Where did the years go?)

Anyway, back to XCP-NG…

There are number of options to ESXi (or vSphere, really). There is XCP-NG, Proxmox VE, Hyper-V, KVM, etc. For a Microsoft shop, Hyper-V would be a logical choice and Hyper-V has come a long way over the years. For a Citrix shop, Citrix Hypervisor (formerly XenServer)) is an obvious option. Proxmox seems really neat. Xen is really cool because it is the basis for XenServer/Citrix Hypervisor (I’ll be sticking with XenServer as “Hypervisor” without the Citrix reads strange.) and XCP-NG with XCP-NG basically being the free, open source version of XenServer.

In the end, I have two potential options and if I’m going open source (free, as in free beer not a free puppy) what are my options? There really seems to be two: XCP-NG or Proxmox. This is a challenge given the maturity of ESXi.

I first tried Proxmox. The interface is really nice. However, after a week of trying to get iSCSI to work (with lots of YouTube videos and Google University) I finally gave up. The problem wasn’t so much with getting iSCSI to work but to work as I want it to work. I want it to work like ESXi where I create a datastore and have all my VMs reside on that datastore. I could easily get a VM to use all the datastore but I wanted a shared datastore; not a LUN (or LUNs) per VM. I tried to get LVM as the filesystem for VM use but for whatever reason I couldn’t get it to work. Basically, after a week I gave up.

The same thing almost happened with XCP-NG. However, after about five days (and more than a little help from Tom Lawrence’s YouTube videos :-)) I have XCP-NG up and running on ye olde DL360 Gen8. I had to replace the onboard P420i RAID controller with my old IBM 1015 in IT mode. The problem with the P420i is not that you cannot change it from RAID to IT mode. Rather, the DL360 Gen8 cannot boot from the P420i when in IT mode. I thought about buying a PCIe M.2 card and using an M.2 SSD I had lying around but I could not get confidence that I would be able to successfully boot from the PCIe card. I thought about finding an HP power/SATA cable but the price on eBay (plus shipping time) was not appealing. I did have to purchase a couple of SAS cables as the HP cables for the P420i weren’t long enough but $25 and 5 days shipping seemed reasonable. The only downside is that the HP drive lights do not work with the non-HP cables. Oh well…

The biggest challenge – as it would be with any transition – is figuring out how X is done on the new platform. I have most things figured out. My feeling is XOA – the management “console” – is not as mature as ESXi. You can go with the XCP version of XOA but without a subscription you cannot get the patches. I decided to go with Roni Väyrynen’s XenOrchestraInstallerUpdater running on Ubuntu (not Debian, because that is what I am familiar with). However, XOA in my opinion is not as mature and polished as ESXi. Maybe the new Xen Orchestra Lite (a/k/a XOA-lite) may fix this. Remember, the web-based ESXi management GUI was pretty rough at one time, too. Time will tell.

I have one “production” VM running on XCP-NG. Using Clonezilla I migrated the VM from ESXi to XCP-NG. Just remember to remove the open-vm-tools first (I use Ubuntu for most of my servers). About the only issue I have found so far is that the network interface names change during the migration from the “ensxxx” format to the “traditional” ethx format. No problem, just update Netplan on first boot.

One thing that I have to figure out is Synology iSCSI and redundancy. The Synology iSCSI for Linux seems to be single path only. I’m sure I’ll figure it out over the next few weeks. I only need another GigE PCIe card for the DL360 as I only have four GigE ports – one for XCP-NG, one for VMs, one for DMZ and one for iSCSI.

I also have to figure out how to deal with 10GigE for any migration. Right now ESXi is directly connected to my RS1221+ so I’ll need another dual port 10GigE port for the DL360 and a 10GigE switch and four DAC cables. That might take time.

Anyway, I do have options…