Mike Pelley's Musings

I used hardware RAID on my HP DL380s via the Smart Array P440ar Controller. As I noted on my previous posts, my VM storage in on the RS1221+ NAS via NFS so ZFS is not a concern. Storage on the server is just for booting Proxmox and maybe some ISO storage.

I would like mange the RAID arrays when I have to replace a drive (future post coming). I found a project on GitHub by mrpeardotnet but that was based on Proxmox 6.x so the instructions are not quite right. Here are my updates:

When you add the repository (being 2025) you need to update the following for TLS and Proxmox 8.3 which uses Debian Bookworm:

echo "deb https://downloads.linux.hpe.com/SDR/repo/mcp bookworm/current non-free" > /etc/apt/sources.list.d/hp-mcp.list

Then add the GPG keys:

curl https://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
curl https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
curl https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key2.pub | apt-key add -

The rest of the mrpearldotnew instructions for the SSA commands are in his GitHub project: https://gist.github.com/mrpeardotnet/a9ce41da99936c0175600f484fa20d03

If you haven’t noticed, Proxmox released its Datacenter Manager back on December 19, 2024 as alpha version 0.1. For an alpha release, it is amazingly stable. So far, it is up to version 01.11. It is not feature complete as the roadmap (https://pve.proxmox.com/wiki/Proxmox_Datacenter_Manager_Roadmap) notes.

That said, one feature that works really well is the migrate function (paper airplane icon highlighted in yellow).

This means you don’t have to set up a Proxmox cluster which is beneficial if you have one or more nodes that are usually shut down for power savings (or noise – my DL360p Gen8 is 1U loud). It does not have the failover you get with a cluster.

One question I have not figured out – with some Google University searching – is what interface is used to migrate the LXC containers or KVM VMs. It seems that it uses the admin (Proxmox VE console) network which is, in my case, 1 GbE rather than my (new) 10 GbE network.

It also doesn’t yet have the ability to add Proxmox Backup Server (which I have also migrated to – as a VM rather than a dedicated server). It is on the roadmap.

If you run more than one Proxmox server, this Proxmox Datacenter Manager is a great move forward. Especially in the enterprise market (i.e., XCP-NG).

After taking a good look at the circuit boards in the 5PX, I was pleasantly surprised to find no leaking or bulging capacitors. I checked a few capacitors in circuit, and they measured acceptable. I “bit the bullet” and purchased four third-party batteries (literally 15% the cost of OEMs from Eaton). I transferred the new batteries paying close attention to how they needed to be wired and everything works just fine. It still seems like a great deal.

Well, I updated the Mini G3 to the latest Proxmox 8.3.3 with proxmox-kernel-6.8.12-7 and it seems that the need to blacklist the tg3 module is gone. In fact, with it blacklisted, you no longer have any networking. Removing tg3 from the blacklist restores networking.

The issue in the previous post, “There seems to be a delay in the network configuration to come fully up and the console show the wrong IP” is gone. This is the result of a bit of stupidity on my part. When I set up Proxmox it uses the default configuration and this provides the IP address from my core network subnet. When I update the network configuration per my previous post, it does not update /etc/hosts. A manual update fixes that. Ooops…

I think that I have finally figured out how to get my EliteDesk 705 G3 Mini PC working. Part of the problem is the Broadcom BCM5762 NIC in which case you have to (a) blacklist the NIC driver tg3 and (b) ensure that you have added GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt" as detailed in my post https://blog.pelleys.com/?p=961.

It seems that you have to blacklist the tg3 driver as the BCM5762 is buggy as all get out – VLANs do not even seem to work (for me, at least) if tg3 is not blacklisted. You seem to need iommu=pt otherwise you will get a “transmit queue 0 timed out” error and lose networking.

It is not perfect. There seems to be a delay in the network configuration to come fully up and the console show the wrong IP. In my case it gets an IP from the native VLAN but then reverts to the set IP, subnet and VLAN.

With that out of the way, I really had to think through the logic of what I was trying to do. Here is the scenario:

• Single NIC
• NFS storage on a different subnet and VLAN
• Need to have the ability for VMs to join different VLANs

The key part of the logic is to remember that NFS needs to be the native bridge interface otherwise NFS tries to connect using the wrong network. Here is an example configuration in/etc/network/interfaces:

auto vmbr0
iface vmbr0 inet static
   address 10.1.1.1/24 ← This is the NFS host IP for the Mini G3
   bridge-ports enp1s0 ← This is the host physical NIC (normal)
   bridge-stp off
   bridge-fd 0
   bridge-vlan-aware yes ← Make sure the bridge is configured for VLANs
   bridge-vids 100 200 300 400 ← I manually restrict to my VM VLANs

# This is the administrative interface and forcing VLAN for it
auto vmbr0.200
iface vmbr0.200 inet static
   address 10.1.200.5/24
   gateway 10.1.200.254

On your switch, UniFi in my case, you need to set the Native VLAN/Network to the network your NFS storage is on (called “NFS-VLAN (100)” in my case) and Tagged VLAN Management to Allow All.

This seems to work even even when using Proxmox Datacenter Manager migration (PDM) of VMs with no transmit queue time out. On that point, even though PDM is currently alpha code, it shows real promise. I like that I do not have to set up a cluster to move VMs between nodes (my HP DL360 Gen8 is usually off – too “whiny” fans and I don’t pay the power bill.

Hope this helps someone!

The home lab journey continues. I was a “good enough” boy last year, so Santa dropped off a Unifi Aggregation Switch. This completes my back end “core” for my standard networking and NFS storage for my Proxmox servers.

I also updated the uplink to the Netgate 6100 to connect to the USW Aggregation at 10 GbE. (One day I will get around to upgrading my Internet link to over 1 Gb/s but based upon usage, that is low on priority list.

A hint of this was my previous post where I was totally confused as to which port was ix1. I always have the first port, ix0 in this case, as my LAN connection.

Here is the basic setup:

I picked up a brush panel as it is a nice way pass the DAC cables through. While I still have a couple of fibre connections between the switches, going forward DAC cables will be the way I will be proceeding unless I have some obscenely long run.

Anyway, here is what the rack looks like as of January 2025:

On a sad note, it looks like a capacitor on the Eaton 5PX 2200 has failed. Once the 5PX goes on battery the unit goes into battery failure, even though the battery is good. It seems to be a fairly “well known” issue, so I have to replace that. No good deal is ever that good, unfortunately.

Hi Folks – an FYI on the Netgate 6100 port assignments. This really messed me up as the SFP port numbering is not clear – at least not clear to me – in the documentation. So, in case anyone else is confused by this, here are the port assignments

• ix3 (SFP and RJ45 shared – 1GbE)
• ix2 (SFP and RJ45 shared – 1GbE)
• ix1 (SFP+ – 10GbE)
• ix0 (SFP+ – 10GbE)

Ok… The truth is I was scrolling through Facebook Marketplace and I saw a an Eaton 5PX2200 UPS for sale. It was rackmount and the pictures looked good. But for an impulse purchase the price was a little high. A few weeks went by and the price dropped a bit – getting closer. Then this weekend it dropped to a price that, even if the batteries were shot, was worth it.

(I should note that the Lenovo is a rebadged Eaton unit. The network module reports Eaton.)

I dropped over and not only did the battery seem to be fine (there was barely enough charge left to check), it looked good (a little pushed in on the back where the power cord comes in), no scratches, and it has the rails (FYI – they only go to 36″ and HP servers need 38″, so too short and I had to use the shelf rails that came with the rack), manuals and CDs. The story the seller told me he took the UPS in trade for some gaming PC parts thinking that using for his gaming PC. Considering any higher end gaming PC draws more than my DL380 Gen9 with dual power supplies that makes sense. Anyway, the Eaton draws 20A which requires the plug below which the seller did not have.

I have a 20A outlet in my home office (future planning 😉 ) so no cord cutting for me.

After moving the DL380 to the Eaton, with the rest of the gear left on the Lenovo RT1.5KVA, here are the runtime (sorry, I didn’t bother to match the brightness and contrast):

Well, that is nice – all ready for winter and any shorter power outages.

Here’s where we are now:

Only thing that is left is my 10GbE switch. And with Winter comes Christmas 🙂

I was going through some old web backups (because of the last post) and found a picture (320×240 and I scaled it up a bit for this post – I think we had a Sony Mavica digital camera).

Here’s some of what I had put in place just before I left for a new job:

• A SAN with Hewlett-Packard FC60/SC10 disks and Brocade Silkworm-based Fibre Channel interconnections
• Hewlett-Packard NetServers
• Open Storage Solutions servers and external disk arrays
• Custom-built Pentium II / III rack-mount servers
• Digital AlphaServer-based network support systems (internal/external DNSes)
• Exabyte 220 tape library system
• Yes, that is a Cisco PIX 1000 (or 500, I can’t actually remember but it used 3.5″ floppies for backups)

Edit: I just realized that there is an IBM workstation running RealProducer (lonely black tower in the middle of the picture) with a whitebox server to the left running RealServer. Those were the days 🙂

On Thursday night, about an hour before going to bed, I decided to do an in-place upgrade to my VPS. I knew I had good backups of WordPress, the websites, etc. so I was not all that concerned. I have done many, many release upgrades of Ubuntu on-site I figured about 30-40 minutes for the upgrade. And indeed, after about 30 minutes the upgrade was finished and it was time to reboot.

I gave the VPS about a minute to restart and… nothing. I could not connect using SSH or using a web browser. I hop onto the VPS console and see a bootloop. Ok, I’ve seen this happen before. Jump into recovery mode and see what was broken and fix it. And it seems I really broke something as I could not even start the recovery console. I have done a couple of in-place release upgrades plus, of course, a bunch of tweaks over the last few years so I can see that causing issues.

Since I was tired I powered off the borked VPS and off to bed I went. I am finally old and wise enough (or maybe my ass was dragging…) to know being tired and being successful often does not work out.

The next day I installed a new instance of the VPS and began the recovery.

The good news:

• The WordPress recovery went fairly easily. The database backup was solid. I just had to remember to create the WordPress database account and grant it access to the WordPress database. Funny about that…
• The recovery of the other websites went fine.
• It gave me a chance to clean up some of the php code, old URLs that had existed for who knows how long, etc.
• Re-implementing Let’s Encrypt SSL was buttery smooth.
• But, more details below…

The bad news:

• I had meant to back up and copy offsite the WordPress content directory. I had part of it done – copy it it over to another directory – but I did not remember to set up the job to copy the contents offsite. Insert Homer Simpson Dunh! here. Then I remembered last month I was messing around with a full copy of my WordPress site locally. Awesome – I have a copy of the WordPress contents. Upload and fixed.
• I somehow did not have an anywhere near current copy of the other websites. That meant that other fixed that I had done over the years did not exist. However, it did mean I had the chance to update the php code.
• I also did not keep a copy of the ufw blocks for, shall we say, less than desirable ISPs. Backed up now and improved by cleanup.

Lesson learned: Make sure your backups are complete. Don’t assume anything.