Mike Pelley's Musings

The home lab journey continues. I was a “good enough” boy last year, so Santa dropped off a Unifi Aggregation Switch. This completes my back end “core” for my standard networking and NFS storage for my Proxmox servers.

I also updated the uplink to the Netgate 6100 to connect to the USW Aggregation at 10 GbE. (One day I will get around to upgrading my Internet link to over 1 Gb/s but based upon usage, that is low on priority list.

A hint of this was my previous post where I was totally confused as to which port was ix1. I always have the first port, ix0 in this case, as my LAN connection.

Here is the basic setup:

I picked up a brush panel as it is a nice way pass the DAC cables through. While I still have a couple of fibre connections between the switches, going forward DAC cables will be the way I will be proceeding unless I have some obscenely long run.

Anyway, here is what the rack looks like as of January 2025:

On a sad note, it looks like a capacitor on the Eaton 5PX 2200 has failed. Once the 5PX goes on battery the unit goes into battery failure, even though the battery is good. It seems to be a fairly “well known” issue, so I have to replace that. No good deal is ever that good, unfortunately.

Hi Folks – an FYI on the Netgate 6100 port assignments. This really messed me up as the SFP port numbering is not clear – at least not clear to me – in the documentation. So, in case anyone else is confused by this, here are the port assignments

• ix3 (SFP and RJ45 shared – 1GbE)
• ix2 (SFP and RJ45 shared – 1GbE)
• ix1 (SFP+ – 10GbE)
• ix0 (SFP+ – 10GbE)

Ok… The truth is I was scrolling through Facebook Marketplace and I saw a an Eaton 5PX2200 UPS for sale. It was rackmount and the pictures looked good. But for an impulse purchase the price was a little high. A few weeks went by and the price dropped a bit – getting closer. Then this weekend it dropped to a price that, even if the batteries were shot, was worth it.

(I should note that the Lenovo is a rebadged Eaton unit. The network module reports Eaton.)

I dropped over and not only did the battery seem to be fine (there was barely enough charge left to check), it looked good (a little pushed in on the back where the power cord comes in), no scratches, and it has the rails (FYI – they only go to 36″ and HP servers need 38″, so too short and I had to use the shelf rails that came with the rack), manuals and CDs. The story the seller told me he took the UPS in trade for some gaming PC parts thinking that using for his gaming PC. Considering any higher end gaming PC draws more than my DL380 Gen9 with dual power supplies that makes sense. Anyway, the Eaton draws 20A which requires the plug below which the seller did not have.

I have a 20A outlet in my home office (future planning 😉 ) so no cord cutting for me.

After moving the DL380 to the Eaton, with the rest of the gear left on the Lenovo RT1.5KVA, here are the runtime (sorry, I didn’t bother to match the brightness and contrast):

Well, that is nice – all ready for winter and any shorter power outages.

Here’s where we are now:

Only thing that is left is my 10GbE switch. And with Winter comes Christmas 🙂

I was going through some old web backups (because of the last post) and found a picture (320×240 and I scaled it up a bit for this post – I think we had a Sony Mavica digital camera).

Here’s some of what I had put in place just before I left for a new job:

• A SAN with Hewlett-Packard FC60/SC10 disks and Brocade Silkworm-based Fibre Channel interconnections
• Hewlett-Packard NetServers
• Open Storage Solutions servers and external disk arrays
• Custom-built Pentium II / III rack-mount servers
• Digital AlphaServer-based network support systems (internal/external DNSes)
• Exabyte 220 tape library system
• Yes, that is a Cisco PIX 1000 (or 500, I can’t actually remember but it used 3.5″ floppies for backups)

Edit: I just realized that there is an IBM workstation running RealProducer (lonely black tower in the middle of the picture) with a whitebox server to the left running RealServer. Those were the days 🙂

On Thursday night, about an hour before going to bed, I decided to do an in-place upgrade to my VPS. I knew I had good backups of WordPress, the websites, etc. so I was not all that concerned. I have done many, many release upgrades of Ubuntu on-site I figured about 30-40 minutes for the upgrade. And indeed, after about 30 minutes the upgrade was finished and it was time to reboot.

I gave the VPS about a minute to restart and… nothing. I could not connect using SSH or using a web browser. I hop onto the VPS console and see a bootloop. Ok, I’ve seen this happen before. Jump into recovery mode and see what was broken and fix it. And it seems I really broke something as I could not even start the recovery console. I have done a couple of in-place release upgrades plus, of course, a bunch of tweaks over the last few years so I can see that causing issues.

Since I was tired I powered off the borked VPS and off to bed I went. I am finally old and wise enough (or maybe my ass was dragging…) to know being tired and being successful often does not work out.

The next day I installed a new instance of the VPS and began the recovery.

The good news:

• The WordPress recovery went fairly easily. The database backup was solid. I just had to remember to create the WordPress database account and grant it access to the WordPress database. Funny about that…
• The recovery of the other websites went fine.
• It gave me a chance to clean up some of the php code, old URLs that had existed for who knows how long, etc.
• Re-implementing Let’s Encrypt SSL was buttery smooth.
• But, more details below…

The bad news:

• I had meant to back up and copy offsite the WordPress content directory. I had part of it done – copy it it over to another directory – but I did not remember to set up the job to copy the contents offsite. Insert Homer Simpson Dunh! here. Then I remembered last month I was messing around with a full copy of my WordPress site locally. Awesome – I have a copy of the WordPress contents. Upload and fixed.
• I somehow did not have an anywhere near current copy of the other websites. That meant that other fixed that I had done over the years did not exist. However, it did mean I had the chance to update the php code.
• I also did not keep a copy of the ufw blocks for, shall we say, less than desirable ISPs. Backed up now and improved by cleanup.

Lesson learned: Make sure your backups are complete. Don’t assume anything.

After years of using desktop UPSes for my server and network gear, I finally bought a rack mount, data centre grade UPS. Over the years, APC was my go-to brand. However, after two of my three UPSes had their batteries die, that all changed especially after buying new batteries – even generic replacements – was approaching the cost of a new UPS.

In searching eBay for the replacement batteries I stumbled across a Canadian (useful due to shipping, import duties when buying from the US) that was selling rerfubished Lenovo RT1.5kVA UPSes with new batteries.

The RT1.5kVA also includes an SNMP network card so I no longer have to use USB connections and can easily have other devices like my pfSense firewall, Proxmox host and two Synology NASes “share” the UPS and not have to rely on some other device.

I now have just under 30 minutes runtime.

The last, just under 3-year-old APC UPS will now be reused for my ISP’s router and ONT. With 1500 watts available that is 5x more than the old 300 watt UPS that is getting rather old and probably needs a new battery, too.

Good bye 2023… Hello 2024! I wish everyone health, wealth and happiness over 2024.

I guess I’m getting a little nostalgic as I have become ye olde IT grey beard…

I thought I would put up some my “server closet” – actually, it was a closet – pictures and tech details from 2002.

The firewall ran SmoothWall 0.9.9. This was a highly modified and hardened VA-Linux kernel that provided firewall, Intruder Detection Systems (IDS) and proxy services.
Tech specs:

• Asus AP53 motherboard
• Pentium 166 MHz CPU
• 128 MB RAM
• One 340 MB IDE hard disk
• 24X E-IDE CD-ROM
• 3Com 3c905B network adaptor
• SMC Ultra network adaptor
• Matrox Millennium G200 PCI video card

The Windows 2000 Server was the file server, secondary internal DNS server and Active Directory Domain Controller. This system was used to run Internet Information Server 5 until it was hacked in the first Code Red attacks in July 2001. It also ran the RealProducer software using a Logitech Color USB WebCam. (I really wish it was an AlphaServer 4100!)
Tech specs:

• Soyo SY-7VBA 133 motherboard
• Celeron 700 MHz CPU
• 192 MB SD-RAM
• Two fixed 9 GB Ultra SCSI disk
• Adaptec AHA-2940U SCSI Controller
• 4/8 GB WangDAT 4mm DAT tape backup
• 48X E-IDE CD-ROM
• ATI Mach64 4 MB PCI video card
• 3Com EtherLink XL 10/100 PCI network card

And last, but not least, the HP NetServer LD Pro that ran Red Hat Linux 7.0. It used Apache with PHP and Perl as the web server. FTP services were handled by ProFTPD. The NetServer also provided primary internal dynamic DNS services with BIND 9. Live Web Cam services are provided by RealServer 8. Of course, it was constantly updated with security patches (after Code Red)!
Tech specs:

• Pentium Pro 180 MHz CPU
• 96 MB ECC RAM
• One fixed 9 GB Ultra SCSI disk
• Two hot swap 9 GB Ultra SCSI disks
• 24X SCSI CD-ROM
• Intel EtherExpress Pro/100 Plus network adaptor