Mike Pelley's Musings

A couple of notes on where I am with my ESXi to Proxmox move.

What I have done:

• I finally bit the bullet and deleted all of my iSCSI LUNs for ESXi – they were simply wasting space at this point.
• I also moved from iSCSI to NFS for my Proxmox VMs. The main difference between iSCSI and NFS is that iSCSI shares data on the block level, and NFS shares data on the file level. Performance is almost the same, but, in some situations, iSCSI can provide better results. For my purposes NFS is fine.
• And NFS has the benefit that I can access the VMs on the NAS in Synology File Station. There is a level of comfort in being able to “see” all of my VMs without having to set up an iSCSI initiator.
• I haven’t worked on network bonding for the NFS 10 GbE connections. Frankly, I do not think I will see any benefits and “added complexity equals added problems.”
• Both my “production” (PROD) Proxmox server (DL380 G9) and my “development” (DEV) Proxmox server (DL360 G8) can access the same NFS shares. That means that my ISOs are shared as well as my Proxmox backups. This also means that I can create a VM on the DEV box and, by backing up the VM to the shared backup location, restore it to the PROD box. While not the smoothest method, it is a rather neat workaround. I do have to be careful about starting any VMs that are visible to both servers, otherwise “bad things” would happen.
• I have moved a number of full VMs to LXC containers. Things like my reverse proxy, a DNS server, Home Assistant (I don’t really have any devices for this yet but with Z-Wave it will give me something to do 🙂 ), and Pi-Hole. There is really no need to have a full blown VM for these simple services. Proxmox makes it easy.
• Pi-Hole – not part of Proxmox (other that it runs in an LXC container) but I finally got around to setting up Pi-Hole. A couple of additional configs were needed as Pi-Hole sits on one subnet while my wired and wireless devices sit on other subnets to get it working for my clients. And, WOW!, the amount of ads, etc. that are blocked are crazy. Here are the last 24 hour stats – 51.4% blocked requests?!?!?!

The To-Do (or think-about) list:

• Adding a Unifi Link Aggregation switch – while both Proxmox nodes are connected directly to the Synology NAS, they cannot connect directly to each other for storage. One node is using the 172.16.10.0/24 subnet and the other 172.168.20.0/24 with one of the two10 GbE port of the NAS assigned to each. With the switch I can have them both on the same subnet (yes, I know that I could have “broken” the 172.16.10/0 subnet in two using /25).
• Adding a “one litre” PC (such as a HP EliteDesk 705 G3) with Proxmox installed and setting up the DL360 and DL380 as a cluster using the 705 to complete the quorum. Allowing the 705 (or whatever) access to storage may be beneficial, but the 705 does not have 10 GbE. I *think* that the main nodes need to have storage on the same subnet to work correctly (I haven’t read up on that, yet). The jury is still out on that given the cost of electricity, heat (not so bad in winter) and noise (the rack sits behind me in my office).

Sometimes when you get to the end of the dock, you want to jump into the water. After getting comfortable (enough) with Proxmox I took the plunge and deleted ESXi 7 from my DL380 Gen9 and moved all of my VMs from the DL360G8 test server. Everything seems to work just fine.

I did not put the change the Smart Array P440ar Controller into IT mode. I left it in HBA mode with one mirror for Proxmox and another mirror for local ISOs. Now, that is a little overkill for the ISOs given they are enterprise SSDs. The other reason why I am not worried about HBA mode is that my VMs are on one of my NASes.

I did leave the (unconverted) ESXi VMs on the NAS and have the ESXi configuration backed up in case I need to revert. I like to have a solid fallback plan…

There is one major last thing I have to sort: I cannot get the bonded 10 GbE connections between the the DL380 and the RS1221+ to be stable. I am pretty sure that this is something with the bonding method I selected. There is no switch between the two – it is just NIC-to-NIC (times two). For now, I will let this sit for awhile.

Overall, I am impressed with Proxmox. VMs seem snappier that under ESXi. There are some quirks: there is no file manager as ESXi (really nice to have), the need to have different storage types for different purposes (head scratch) and the fact if you use LVM for iSCSI you cannot take snapshots (what?!?!).

I took the plunge over the last few nights and migrated all of my VMs from ESXi to Proxmox. Outside of the previously mentioned updates to the NIC configurations, everything went fine. You will need to follow the instructions for Windows guests per the instructions (e.g., make sure you manually modify the Windows’ pve file and change scsi0 to sata0). Once you do that, you will be able to boot successfully. You will also have to add the virtio drivers as well. This is a manual download outside of Proxmox.

I have the ESXi server shutdown so I can restart that if necessary. The only thing I will have to do if I need to go back to ESXi is to get the weather station data.

To do list:

• Research how to use the 10GbE links working between the DL380G9 and the RS1221+. I’ll be taking my time with this one as I do not want to change the ESXi server to Proxmox until I have some more time with Proxmox and feel comfortable.
• Get syslog sending to my DS216+II. I just started looking into this and it seems that, unlike ESXi, I need to do this at the underlying Proxmox Debian 11 operating system. Remember that ESXi (and XCP-NG) are Type 1 hypervisors and Proxmox is a Type 2 hypervisor.

After about a couple of weeks trying out XCP-NG I do not think it is for me. I still think that XCP-NG has great capabilities but it does not really seem polished. Maybe I needed a couple more weeks to get more familiar with XCP-NG but I really wasn’t feeling it. I know that it took some time for me to become confident with ESXi – and that started with ESXi 5 – but XCP-NG seems so “difficult.”

So, back to Proxmox. After a couple of evenings working on my LVM on iSCSI problem, I finally figured it out. The problem I was having with iSCSI was that my ESXi server also had access to host initiator along with Proxmox. The Proxmox host initiator configuration was set up as Linux which does not allow sharing and the ESXi host configuration was set up as VMware (of course) which allows sharing. So while Proxmox could see the target Synology (it appears, anyway) would not allow both ESXi and Proxmox to share the target. Once I removed ESXi’s access to the new target connectivity was fine.

The next part was to add an LVM (it could have been ZFS but I wanted to keep it simple) so that multiple VMs could reside on the iSCSI share. The last time I tried Proxmox I got iSCSI to work but it would have been a separate LUN for each VM’s disks and that would have been a pain. I can see uses for a LUN dedicated to a VM (security, performance, etc.) but I don’t need that for my home lab. I finally realized that I manually had to add the LUN’s Volume Group to the LVM configuration. I think that this can automatically populate but it didn’t work for me. Once this had that figured out, I could create VMs on the iSCSI LVM space.

I also took a test VM I created on ESXi (just a simple Ubuntu 20.04.4 LTS install) as a migration test. I exported that VM as an OVF and copied the files to a temp space on Proxmox. Then running qm importovf I imported the test VM. Proxmox’s migration instructions were dead on. No Clonezilla in the mix. Nothing against Clonezilla – it is a excellent package – but this seems much more simple to me.

I even forgot to remove the open-vm-tools on the test VM prior to exporting/importing. No problem, just remove the package. Networking, of course, had to be reconfigured. First, there was no network adaptor but that was expected given the type of virtual NIC changed. I just had to add a NIC to the VM’s configuration and reboot. Second, I had to update netplan as the new NIC had a different device ID. Update the YAML file with the right device ID and restart netplan. Bang, DHCP brings back an IP and you have network access. I then installed the qemu-guest-agent and reboot. VM migrated.

Next steps: 1. I’m going to leave the couple of VMs running for a few days to check stability. The (production) VM I migrated as a test to XCP-NG had networking issues after an hour or so that I couldn’t resolve. It could have been me, but still it was frustrating. 2. I have to dig deeper into redundant iSCSI. I have my dual 10GbE connectivity between ESXi and Synology but I need to figure out any gotchas. If this pans out, moving to Proxmox won’t be all that painful!

After far too many years it looks like ArmA III is working on Linux. While you still have to use Steam’s Proton Expermental branch performance matches, for me at least, Windows 10.

My desktop PC is an old i7-4790K, Asus B85M-D Plus, Nvidia GTX 1070, 2 x 8GB DDR3 RAM, and a Samsung EVO 850 for Windows and and an EVO 870 for Ubuntu. The framerates are the same under both Windows 10 and Ubuntu 22.04 – 60 FPS. I think this limit is due to my monitors.

The only issue is that is I had two – two! – Kingston A400 480GB drives that were D.O.A. They would not show up in the BIOS as Kingston drives but as Phison PS3111-S11 with 21MB (yes, MB) of space (which I think is the cache). I tried them on two different motherboards with the same results. I bought the first one back to the store (they only had two in stock) and had no problems exchanging it for another A400. However, when I got back home I had the same issue. Given there were only two in stock, I suspect that they are from the same batch. It has been some time since I had problems with drives from the same batch being wanky – the last time was with two Seagate IronWolf 4TB drives. So, it happens.

When I returned the second A400 I went with a Samsung EVO – all the rest of my SSDs (SATA III and M.2 PCIe) are Samsung so the extra $40 seems worth it. (The only exception is there are the 4 Hitachi Ultrastar SSD400M Enterprise SSD in my DL380 Gen9 – but they are enterprise drives :-)). Anyway, put in the Samsung and the install went more or less flawlessly.

I say more or less flawlessly because I always have at least two paritions – one for / and another for /home and I forgot to assign /home during the install. No big deal, just a pain in the butt to move /home to its own partition. Oh, and I had installed ArmA so there was about 140GB of ArmA, Steam and mods to move…

Once my partner is back from vacation, we’ll do some missions to see if this works as well as I think it will.

It is so nice to ditch Windows!

Given the pending Broadcom acquisition of VMware, I have been giving some thought about an alternative for my homelab virtualisation solution. Given Broadcom’s previous approaches to acquisitions, given I am using the free version of ESXi I have concerns whether the free version will continue.

These concerns are more than just for the homelab user. Organisations who are VMware customers are also having concerns. As Ed Scannell of TechTarget SearchDataCenter noted in his article Broadcom’s acquire-and-axe history concerns VMware users (retrieved 2022 July 10): “Broadcom’s history: Acquire and axe”. This make perfect business sense: sell off or stop the unprofitable/low profitability products, concentrate on your top customers and maximize shareholder return. That is your fiduciary responsibility.

However, that doesn’t help the homelab user. (Let alone the SMB market…) How this all plays out, well time will tell.

I’ve been a big advocate of VMware starting back in the very early 2000’s. Back in the days of VMware GSX (basically what VMware Workstation is now) and ESX. Back then, looking at server utilisation where everything was physical showed a massive underutilization. “Old fogies” like me will remember your NetServers and Proliants with average CPU utilizations around 25% (or 5-10%…).

Basically, I have been using VMware ESX and later ESXi both professionally and personally for about 20 years. (Wow! Where did the years go?)

Anyway, back to XCP-NG…

There are number of options to ESXi (or vSphere, really). There is XCP-NG, Proxmox VE, Hyper-V, KVM, etc. For a Microsoft shop, Hyper-V would be a logical choice and Hyper-V has come a long way over the years. For a Citrix shop, Citrix Hypervisor (formerly XenServer)) is an obvious option. Proxmox seems really neat. Xen is really cool because it is the basis for XenServer/Citrix Hypervisor (I’ll be sticking with XenServer as “Hypervisor” without the Citrix reads strange.) and XCP-NG with XCP-NG basically being the free, open source version of XenServer.

In the end, I have two potential options and if I’m going open source (free, as in free beer not a free puppy) what are my options? There really seems to be two: XCP-NG or Proxmox. This is a challenge given the maturity of ESXi.

I first tried Proxmox. The interface is really nice. However, after a week of trying to get iSCSI to work (with lots of YouTube videos and Google University) I finally gave up. The problem wasn’t so much with getting iSCSI to work but to work as I want it to work. I want it to work like ESXi where I create a datastore and have all my VMs reside on that datastore. I could easily get a VM to use all the datastore but I wanted a shared datastore; not a LUN (or LUNs) per VM. I tried to get LVM as the filesystem for VM use but for whatever reason I couldn’t get it to work. Basically, after a week I gave up.

The same thing almost happened with XCP-NG. However, after about five days (and more than a little help from Tom Lawrence’s YouTube videos :-)) I have XCP-NG up and running on ye olde DL360 Gen8. I had to replace the onboard P420i RAID controller with my old IBM 1015 in IT mode. The problem with the P420i is not that you cannot change it from RAID to IT mode. Rather, the DL360 Gen8 cannot boot from the P420i when in IT mode. I thought about buying a PCIe M.2 card and using an M.2 SSD I had lying around but I could not get confidence that I would be able to successfully boot from the PCIe card. I thought about finding an HP power/SATA cable but the price on eBay (plus shipping time) was not appealing. I did have to purchase a couple of SAS cables as the HP cables for the P420i weren’t long enough but $25 and 5 days shipping seemed reasonable. The only downside is that the HP drive lights do not work with the non-HP cables. Oh well…

The biggest challenge – as it would be with any transition – is figuring out how X is done on the new platform. I have most things figured out. My feeling is XOA – the management “console” – is not as mature as ESXi. You can go with the XCP version of XOA but without a subscription you cannot get the patches. I decided to go with Roni Väyrynen’s XenOrchestraInstallerUpdater running on Ubuntu (not Debian, because that is what I am familiar with). However, XOA in my opinion is not as mature and polished as ESXi. Maybe the new Xen Orchestra Lite (a/k/a XOA-lite) may fix this. Remember, the web-based ESXi management GUI was pretty rough at one time, too. Time will tell.

I have one “production” VM running on XCP-NG. Using Clonezilla I migrated the VM from ESXi to XCP-NG. Just remember to remove the open-vm-tools first (I use Ubuntu for most of my servers). About the only issue I have found so far is that the network interface names change during the migration from the “ensxxx” format to the “traditional” ethx format. No problem, just update Netplan on first boot.

One thing that I have to figure out is Synology iSCSI and redundancy. The Synology iSCSI for Linux seems to be single path only. I’m sure I’ll figure it out over the next few weeks. I only need another GigE PCIe card for the DL360 as I only have four GigE ports – one for XCP-NG, one for VMs, one for DMZ and one for iSCSI.

I also have to figure out how to deal with 10GigE for any migration. Right now ESXi is directly connected to my RS1221+ so I’ll need another dual port 10GigE port for the DL360 and a 10GigE switch and four DAC cables. That might take time.

Anyway, I do have options…

I’ve been running iSCSI between my DL380 Gen9 VMware ESXi 7.0.3 server and Synology RS1221+ NAS at 1GbE. It works okay but as one would expect it is not all that quick. For my birthday I received a Synology E10G21-F2 10GbE dual SFP+ Port adaptor:

The DL380 has a HP FlexFabric 10Gb 2-port 554FLR-SFP Adapter that I got for Christmas:

While I have used OM3 fibre on my UniFi switches, this time I decided to go with DAC cables. Fibre is, well, neat but DAC is simpler with no issues of dirty fibre connectors, etc. Probably a little cheaper as well. I have redundant connections between the ESXi server and the NAS.

The start up of the VMs seems a little faster to start but is a little hard to judge. A Windows Server 2022 VM using CrystalDiskMark showed the following results:

On the Synology NAS, here’s the results:

Note that these scores are in megabytes per second, so we need to multiply by 8:
* Windows Server: 5,678.64 megabits per second
* Synology NAS: 5,388.8 megabits per second
Not bad with the overhead of a VM.

Note that there is no switch involved. I have a direct connection between ESXi and the NAS. Eventually I may add a 10 GbE switch (likely a UniFi USW-Aggregation) and a 10 GbE card to the old DL360 Gen8. I can still connect the DL360 to the iSCSI targets using 1 GbE. The DL360 is only for testing anyway.

I decided to treat myself. It was definitely not the way I wanted to treat myself as it was from my inheritance from the passing of my father in October 2020. But, after my sister and myself got the estate straightened away – no issues but it is a great deal of work – my sister’s advice to do something for myself made sense.

So… My “treat” is an HP DL380 Gen9.

The specs are:

• 2 x Xeon E5-2687W V3 CPUs (25 MB cache, 3.10 GHz/3.50 GHz boost, 10 cores/20 threads)
• 128GB DDR4 Registered ECC RAM
• 4 x 400GB SSD SAS  2.5” drives
• HP SmartArray P440AR RAID Controller  with FBWC
• 4 X GbE network ports
• 2 x 10 GbE via a HP FlexibleLOM FlexFabric 554FLR-SFP+ adapter

The DL360 Gen8 is a fine server but it is a little old. It also is rather loud when a fan fails. I think all 1U servers can have loud fans. The DL380 is 2U and the fans are much quieter.

The dual 10 GbE LOM card was icing on the cake. I run my VMs on the SAN although the four Hitachi Ultrastar SSD400M drives are mighty fast on the 12 Gbit/s SAS backplane (yes, they are a little old – more info here). My Synology RS1221+ has lots of storage – not super fast but good enough for a home lab. I use iSCSI (see this post for the background) and 1 GbE can be sluggish. I was planning on move to 10 GbE but I needed two dual port PCIe cards (one for the RS1221+ and the other for the server) and the cards needed to be compatible with Synology DSM and VMware ESXi 7.0. That was going to be a little pricey. I now just need the dual port card for the RS1221+ and a couple of DAC cables.

So far, I have tried out Proxmox 7 and ESXi 7. ESXi is an old friend – it just works. Proxmox is rather interesting. It has some fine features. I’m not quite sold on the different locations and file types that different file locations can use. I haven’t taken the time to find out why and there probably is a reason. Still, it is odd. Being open source is nice. I like having the source code open to independent audit. That said, ESXi is used by some, shall we say, very security conscious institutions. Like other open source projects like pfSense and TruNAS the devices supported always seems to be much broader and older hardware remains supported. I did have to switch P440AR controller to IT mode. I really like that feature – real IT mode with cache turned off. It does not seem to be a bunch of individual “RAID-0” arrays. ZFS could see all the SMART info. All said, I can appreciate those who endorse Proxmox.

This was the first time I experimented with a shared iSCSI LUN using VMFS. VMFS 6 is cluster aware so multiple ESXi hosts can share the storage. I simply had to allow the iSCSI LUNs on the Synology box multiple access and give permission to both the DL360 and DL380 to mount the LUNs. All you have to do is to register the ESXi VM on the new node and Bob’s your uncle. I did make sure that the network configuration and names are the same on both the DL360 and DL380. I did change the P440AR to IR mode and create a RAID-5 configuration. Yes, ESXi works just fine too.

Of note, the iSCSI issues that I had and wrote about previously did not occur at all – despite my trying to make it happen. This could be a result of either the latest ESXi update or the Synology DSM update or both.

I might give XCP-NG another try. My last attempt did not impress me. Not that XCP-NG isn’t any good – it is. It is just the way it does things. Maybe it is me but I just can’t get my head around XCP-NG.

Anyway, I won’t be making a decision on how I’ll be proceeding. I need to add the System Insight Display (SID).

Strange why it isn’t standard. That treat will be opened December 25th.