Mike Pelley's Musings

After about a couple of weeks trying out XCP-NG I do not think it is for me. I still think that XCP-NG has great capabilities but it does not really seem polished. Maybe I needed a couple more weeks to get more familiar with XCP-NG but I really wasn’t feeling it. I know that it took some time for me to become confident with ESXi – and that started with ESXi 5 – but XCP-NG seems so “difficult.”

So, back to Proxmox. After a couple of evenings working on my LVM on iSCSI problem, I finally figured it out. The problem I was having with iSCSI was that my ESXi server also had access to host initiator along with Proxmox. The Proxmox host initiator configuration was set up as Linux which does not allow sharing and the ESXi host configuration was set up as VMware (of course) which allows sharing. So while Proxmox could see the target Synology (it appears, anyway) would not allow both ESXi and Proxmox to share the target. Once I removed ESXi’s access to the new target connectivity was fine.

The next part was to add an LVM (it could have been ZFS but I wanted to keep it simple) so that multiple VMs could reside on the iSCSI share. The last time I tried Proxmox I got iSCSI to work but it would have been a separate LUN for each VM’s disks and that would have been a pain. I can see uses for a LUN dedicated to a VM (security, performance, etc.) but I don’t need that for my home lab. I finally realized that I manually had to add the LUN’s Volume Group to the LVM configuration. I think that this can automatically populate but it didn’t work for me. Once this had that figured out, I could create VMs on the iSCSI LVM space.

I also took a test VM I created on ESXi (just a simple Ubuntu 20.04.4 LTS install) as a migration test. I exported that VM as an OVF and copied the files to a temp space on Proxmox. Then running qm importovf I imported the test VM. Proxmox’s migration instructions were dead on. No Clonezilla in the mix. Nothing against Clonezilla – it is a excellent package – but this seems much more simple to me.

I even forgot to remove the open-vm-tools on the test VM prior to exporting/importing. No problem, just remove the package. Networking, of course, had to be reconfigured. First, there was no network adaptor but that was expected given the type of virtual NIC changed. I just had to add a NIC to the VM’s configuration and reboot. Second, I had to update netplan as the new NIC had a different device ID. Update the YAML file with the right device ID and restart netplan. Bang, DHCP brings back an IP and you have network access. I then installed the qemu-guest-agent and reboot. VM migrated.

Next steps: 1. I’m going to leave the couple of VMs running for a few days to check stability. The (production) VM I migrated as a test to XCP-NG had networking issues after an hour or so that I couldn’t resolve. It could have been me, but still it was frustrating. 2. I have to dig deeper into redundant iSCSI. I have my dual 10GbE connectivity between ESXi and Synology but I need to figure out any gotchas. If this pans out, moving to Proxmox won’t be all that painful!

After far too many years it looks like ArmA III is working on Linux. While you still have to use Steam’s Proton Expermental branch performance matches, for me at least, Windows 10.

My desktop PC is an old i7-4790K, Asus B85M-D Plus, Nvidia GTX 1070, 2 x 8GB DDR3 RAM, and a Samsung EVO 850 for Windows and and an EVO 870 for Ubuntu. The framerates are the same under both Windows 10 and Ubuntu 22.04 – 60 FPS. I think this limit is due to my monitors.

The only issue is that is I had two – two! – Kingston A400 480GB drives that were D.O.A. They would not show up in the BIOS as Kingston drives but as Phison PS3111-S11 with 21MB (yes, MB) of space (which I think is the cache). I tried them on two different motherboards with the same results. I bought the first one back to the store (they only had two in stock) and had no problems exchanging it for another A400. However, when I got back home I had the same issue. Given there were only two in stock, I suspect that they are from the same batch. It has been some time since I had problems with drives from the same batch being wanky – the last time was with two Seagate IronWolf 4TB drives. So, it happens.

When I returned the second A400 I went with a Samsung EVO – all the rest of my SSDs (SATA III and M.2 PCIe) are Samsung so the extra $40 seems worth it. (The only exception is there are the 4 Hitachi Ultrastar SSD400M Enterprise SSD in my DL380 Gen9 – but they are enterprise drives :-)). Anyway, put in the Samsung and the install went more or less flawlessly.

I say more or less flawlessly because I always have at least two paritions – one for / and another for /home and I forgot to assign /home during the install. No big deal, just a pain in the butt to move /home to its own partition. Oh, and I had installed ArmA so there was about 140GB of ArmA, Steam and mods to move…

Once my partner is back from vacation, we’ll do some missions to see if this works as well as I think it will.

It is so nice to ditch Windows!

Given the pending Broadcom acquisition of VMware, I have been giving some thought about an alternative for my homelab virtualisation solution. Given Broadcom’s previous approaches to acquisitions, given I am using the free version of ESXi I have concerns whether the free version will continue.

These concerns are more than just for the homelab user. Organisations who are VMware customers are also having concerns. As Ed Scannell of TechTarget SearchDataCenter noted in his article Broadcom’s acquire-and-axe history concerns VMware users (retrieved 2022 July 10): “Broadcom’s history: Acquire and axe”. This make perfect business sense: sell off or stop the unprofitable/low profitability products, concentrate on your top customers and maximize shareholder return. That is your fiduciary responsibility.

However, that doesn’t help the homelab user. (Let alone the SMB market…) How this all plays out, well time will tell.

I’ve been a big advocate of VMware starting back in the very early 2000’s. Back in the days of VMware GSX (basically what VMware Workstation is now) and ESX. Back then, looking at server utilisation where everything was physical showed a massive underutilization. “Old fogies” like me will remember your NetServers and Proliants with average CPU utilizations around 25% (or 5-10%…).

Basically, I have been using VMware ESX and later ESXi both professionally and personally for about 20 years. (Wow! Where did the years go?)

Anyway, back to XCP-NG…

There are number of options to ESXi (or vSphere, really). There is XCP-NG, Proxmox VE, Hyper-V, KVM, etc. For a Microsoft shop, Hyper-V would be a logical choice and Hyper-V has come a long way over the years. For a Citrix shop, Citrix Hypervisor (formerly XenServer)) is an obvious option. Proxmox seems really neat. Xen is really cool because it is the basis for XenServer/Citrix Hypervisor (I’ll be sticking with XenServer as “Hypervisor” without the Citrix reads strange.) and XCP-NG with XCP-NG basically being the free, open source version of XenServer.

In the end, I have two potential options and if I’m going open source (free, as in free beer not a free puppy) what are my options? There really seems to be two: XCP-NG or Proxmox. This is a challenge given the maturity of ESXi.

I first tried Proxmox. The interface is really nice. However, after a week of trying to get iSCSI to work (with lots of YouTube videos and Google University) I finally gave up. The problem wasn’t so much with getting iSCSI to work but to work as I want it to work. I want it to work like ESXi where I create a datastore and have all my VMs reside on that datastore. I could easily get a VM to use all the datastore but I wanted a shared datastore; not a LUN (or LUNs) per VM. I tried to get LVM as the filesystem for VM use but for whatever reason I couldn’t get it to work. Basically, after a week I gave up.

The same thing almost happened with XCP-NG. However, after about five days (and more than a little help from Tom Lawrence’s YouTube videos :-)) I have XCP-NG up and running on ye olde DL360 Gen8. I had to replace the onboard P420i RAID controller with my old IBM 1015 in IT mode. The problem with the P420i is not that you cannot change it from RAID to IT mode. Rather, the DL360 Gen8 cannot boot from the P420i when in IT mode. I thought about buying a PCIe M.2 card and using an M.2 SSD I had lying around but I could not get confidence that I would be able to successfully boot from the PCIe card. I thought about finding an HP power/SATA cable but the price on eBay (plus shipping time) was not appealing. I did have to purchase a couple of SAS cables as the HP cables for the P420i weren’t long enough but $25 and 5 days shipping seemed reasonable. The only downside is that the HP drive lights do not work with the non-HP cables. Oh well…

The biggest challenge – as it would be with any transition – is figuring out how X is done on the new platform. I have most things figured out. My feeling is XOA – the management “console” – is not as mature as ESXi. You can go with the XCP version of XOA but without a subscription you cannot get the patches. I decided to go with Roni Väyrynen’s XenOrchestraInstallerUpdater running on Ubuntu (not Debian, because that is what I am familiar with). However, XOA in my opinion is not as mature and polished as ESXi. Maybe the new Xen Orchestra Lite (a/k/a XOA-lite) may fix this. Remember, the web-based ESXi management GUI was pretty rough at one time, too. Time will tell.

I have one “production” VM running on XCP-NG. Using Clonezilla I migrated the VM from ESXi to XCP-NG. Just remember to remove the open-vm-tools first (I use Ubuntu for most of my servers). About the only issue I have found so far is that the network interface names change during the migration from the “ensxxx” format to the “traditional” ethx format. No problem, just update Netplan on first boot.

One thing that I have to figure out is Synology iSCSI and redundancy. The Synology iSCSI for Linux seems to be single path only. I’m sure I’ll figure it out over the next few weeks. I only need another GigE PCIe card for the DL360 as I only have four GigE ports – one for XCP-NG, one for VMs, one for DMZ and one for iSCSI.

I also have to figure out how to deal with 10GigE for any migration. Right now ESXi is directly connected to my RS1221+ so I’ll need another dual port 10GigE port for the DL360 and a 10GigE switch and four DAC cables. That might take time.

Anyway, I do have options…

I’ve been running iSCSI between my DL380 Gen9 VMware ESXi 7.0.3 server and Synology RS1221+ NAS at 1GbE. It works okay but as one would expect it is not all that quick. For my birthday I received a Synology E10G21-F2 10GbE dual SFP+ Port adaptor:

The DL380 has a HP FlexFabric 10Gb 2-port 554FLR-SFP Adapter that I got for Christmas:

While I have used OM3 fibre on my UniFi switches, this time I decided to go with DAC cables. Fibre is, well, neat but DAC is simpler with no issues of dirty fibre connectors, etc. Probably a little cheaper as well. I have redundant connections between the ESXi server and the NAS.

The start up of the VMs seems a little faster to start but is a little hard to judge. A Windows Server 2022 VM using CrystalDiskMark showed the following results:

On the Synology NAS, here’s the results:

Note that these scores are in megabytes per second, so we need to multiply by 8:
* Windows Server: 5,678.64 megabits per second
* Synology NAS: 5,388.8 megabits per second
Not bad with the overhead of a VM.

Note that there is no switch involved. I have a direct connection between ESXi and the NAS. Eventually I may add a 10 GbE switch (likely a UniFi USW-Aggregation) and a 10 GbE card to the old DL360 Gen8. I can still connect the DL360 to the iSCSI targets using 1 GbE. The DL360 is only for testing anyway.

I decided to treat myself. It was definitely not the way I wanted to treat myself as it was from my inheritance from the passing of my father in October 2020. But, after my sister and myself got the estate straightened away – no issues but it is a great deal of work – my sister’s advice to do something for myself made sense.

So… My “treat” is an HP DL380 Gen9.

The specs are:

• 2 x Xeon E5-2687W V3 CPUs (25 MB cache, 3.10 GHz/3.50 GHz boost, 10 cores/20 threads)
• 128GB DDR4 Registered ECC RAM
• 4 x 400GB SSD SAS  2.5” drives
• HP SmartArray P440AR RAID Controller  with FBWC
• 4 X GbE network ports
• 2 x 10 GbE via a HP FlexibleLOM FlexFabric 554FLR-SFP+ adapter

The DL360 Gen8 is a fine server but it is a little old. It also is rather loud when a fan fails. I think all 1U servers can have loud fans. The DL380 is 2U and the fans are much quieter.

The dual 10 GbE LOM card was icing on the cake. I run my VMs on the SAN although the four Hitachi Ultrastar SSD400M drives are mighty fast on the 12 Gbit/s SAS backplane (yes, they are a little old – more info here). My Synology RS1221+ has lots of storage – not super fast but good enough for a home lab. I use iSCSI (see this post for the background) and 1 GbE can be sluggish. I was planning on move to 10 GbE but I needed two dual port PCIe cards (one for the RS1221+ and the other for the server) and the cards needed to be compatible with Synology DSM and VMware ESXi 7.0. That was going to be a little pricey. I now just need the dual port card for the RS1221+ and a couple of DAC cables.

So far, I have tried out Proxmox 7 and ESXi 7. ESXi is an old friend – it just works. Proxmox is rather interesting. It has some fine features. I’m not quite sold on the different locations and file types that different file locations can use. I haven’t taken the time to find out why and there probably is a reason. Still, it is odd. Being open source is nice. I like having the source code open to independent audit. That said, ESXi is used by some, shall we say, very security conscious institutions. Like other open source projects like pfSense and TruNAS the devices supported always seems to be much broader and older hardware remains supported. I did have to switch P440AR controller to IT mode. I really like that feature – real IT mode with cache turned off. It does not seem to be a bunch of individual “RAID-0” arrays. ZFS could see all the SMART info. All said, I can appreciate those who endorse Proxmox.

This was the first time I experimented with a shared iSCSI LUN using VMFS. VMFS 6 is cluster aware so multiple ESXi hosts can share the storage. I simply had to allow the iSCSI LUNs on the Synology box multiple access and give permission to both the DL360 and DL380 to mount the LUNs. All you have to do is to register the ESXi VM on the new node and Bob’s your uncle. I did make sure that the network configuration and names are the same on both the DL360 and DL380. I did change the P440AR to IR mode and create a RAID-5 configuration. Yes, ESXi works just fine too.

Of note, the iSCSI issues that I had and wrote about previously did not occur at all – despite my trying to make it happen. This could be a result of either the latest ESXi update or the Synology DSM update or both.

I might give XCP-NG another try. My last attempt did not impress me. Not that XCP-NG isn’t any good – it is. It is just the way it does things. Maybe it is me but I just can’t get my head around XCP-NG.

Anyway, I won’t be making a decision on how I’ll be proceeding. I need to add the System Insight Display (SID).

Strange why it isn’t standard. That treat will be opened December 25th.

Fall is fully here. We’ve had our first snowfall. About 15-20 cm November 5th to 6th overnight. Most of it has melted – but not all. I actually put my snow tires on…

Anyway, I’ve been a little bored so…

Since my last post, there have been some developments. Booting from USB went from being inconsistent to being a non-starter. So, back to the old faithful hard drive boot. I have four internal 300GB SAS drives in a RAID-5 configuration. Since I have my VMs on the Synology NAS using iSCSI, the internal drive array was never used for much. Sometimes I would have a specific VM that wanted to make sure would be up in the event that the NAS went down such as the UniFi controller. I realised that isn’t a real worry. Along with having backups on both NASes I have an offline backup where the configurations of pfSense, Unifi and ESXi are kept.

ESXi backups are important as noted in previous posts. For quick reference (maybe mine!) the instructions are provided by VMware in the KB article How to back up ESXi host configuration. More steps are needed than pfSense of UniFi but it is something I do on at least a monthly basis or before and after I make any major configuration changes.

I’m not sure if booting ESXi from USB is an issue with my DL360p Gen8 server, the USB thumb drive (and I tried several) or ESXi 7.x. I do know that booting from the internal drive array does work and I’ll stick with that.

I also updated to the most current patch level of ESXi after the change to booting from the internal drive array. The update to ESXi 7.0.2 (build number 17867351) went flawlessly – although as slow as always. A great resource (especially if you do not use vSphere) is VMware Front Experience‘s VMware ESXi Patch Tracker. Great resource – including step-by-step instructions in a pop-up.

I realized last night that in my efforts to get ESXi up and running, I had installed the original release of ESXi 7.0. Since I had taken Friday off work to take care of a few things that needed to be done during the weekdays I decided to upgrade ESXi to Update 2. It also allowed me to test my theory from my previous post.

The update worked as planned (and boy-oh-boy are updates slow!) but was successful. As I expected the iSCSI share did not show up in ESXi. Let’s check the NAS and see if the targets are still there and if the ESXi server is connected. Yep, ESXi is shown as a host in DSM.

Okay, ssh into the ESXi box. Can I ping the NAS? Yes, working fine.

Log into the ESXi console (again, I’m not using vSphere). The software iSCSI sees the host and the target. But, no iSCSI devices are showing up under storage devices and obviously the datastores are not present.

Let’s try my theory from the last post. Try esxcfg-volume -l and get the UUIDs. Well now, iSCSI volumes are not showing up. Let’s try a few more pings to the NAS, rescan the devices. Nope, not working.

Time for some deep thinking. No panic this time as I know that the VMs are still in the iSCSI LUNs and I just have to get them mounted. And I have good backups!

Some more Googling and the first result from Reddit (vSphere 7.0 U2 iSCSI not working with older HP Lefthand SAN) seems to have the answer. It seems that the issue is around the IQN and how ESXi is handling this. I deleted the ESXi host on the NAS and re-created it. A rescan of the storage devices showed the iSCSI LUNs. I then did the esxcfg-volume -l and mounted, using esxcfg-volume -M UUID (with the “M” rather than the “m” option), the two UUIDs. Bang, the datastores reappears. While the article is for upgrades and not reboots, my gut is that this problem will persist until the bug is squashed. I haven’t tested a reboot because this is a pain in the arse and I have other things I have to do.

From the Reddit post, there is now a KB article from VMware on this issue (iSCSI adapter IQN may change during the upgrade of ESXi 7.0 U1 (84339)). I didn’t see the workaround (I’ll try it if a patch not available and I have to shutdown the ESXi or NAS) but here is that workaround example from the article:

To work around the issue:
Prior to the upgrade, use the esxcli get and set commands to set the generated iSCSI adapter IQN explicitly. As the IQN is a user setting it won’t change after the upgrade.
Get the IQN details:
$ esxcli iscsi adapter get -A vmhba67

vmhba67
   Name: iqn.1998-01.com.vmware:w1-hs3-n2503.eng.vmware.com:452738760:67

Set the IQN details:
$ esxcli iscsi adapter set -A vmhba67 -n iqn.1998-01.com.vmware:w1-hs3-n2503.eng.vmware.com:452738760:67

EDIT

I couldn’t wait for the next time that I needed to reboot the ESXi server. That is probably a good idea given that I probably don’t want to have a number of things going on at the same time. I applied the VMware KB work around and then shutdown the VMs and rebooted the server.

SUCCESS! The iSCSI datastores automatically came up. A gin and tonic with lots of ice may be in order!