Mike Pelley's Musings

Ok… The truth is I was scrolling through Facebook Marketplace and I saw a an Eaton 5PX2200 UPS for sale. It was rackmount and the pictures looked good. But for an impulse purchase the price was a little high. A few weeks went by and the price dropped a bit – getting closer. Then this weekend it dropped to a price that, even if the batteries were shot, was worth it.

(I should note that the Lenovo is a rebadged Eaton unit. The network module reports Eaton.)

I dropped over and not only did the battery seem to be fine (there was barely enough charge left to check), it looked good (a little pushed in on the back where the power cord comes in), no scratches, and it has the rails (FYI – they only go to 36″ and HP servers need 38″, so too short and I had to use the shelf rails that came with the rack), manuals and CDs. The story the seller told me he took the UPS in trade for some gaming PC parts thinking that using for his gaming PC. Considering any higher end gaming PC draws more than my DL380 Gen9 with dual power supplies that makes sense. Anyway, the Eaton draws 20A which requires the plug below which the seller did not have.

I have a 20A outlet in my home office (future planning 😉 ) so no cord cutting for me.

After moving the DL380 to the Eaton, with the rest of the gear left on the Lenovo RT1.5KVA, here are the runtime (sorry, I didn’t bother to match the brightness and contrast):

Well, that is nice – all ready for winter and any shorter power outages.

Here’s where we are now:

Only thing that is left is my 10GbE switch. And with Winter comes Christmas 🙂

I was going through some old web backups (because of the last post) and found a picture (320×240 and I scaled it up a bit for this post – I think we had a Sony Mavica digital camera).

Here’s some of what I had put in place just before I left for a new job:

• A SAN with Hewlett-Packard FC60/SC10 disks and Brocade Silkworm-based Fibre Channel interconnections
• Hewlett-Packard NetServers
• Open Storage Solutions servers and external disk arrays
• Custom-built Pentium II / III rack-mount servers
• Digital AlphaServer-based network support systems (internal/external DNSes)
• Exabyte 220 tape library system
• Yes, that is a Cisco PIX 1000 (or 500, I can’t actually remember but it used 3.5″ floppies for backups)

Edit: I just realized that there is an IBM workstation running RealProducer (lonely black tower in the middle of the picture) with a whitebox server to the left running RealServer. Those were the days 🙂

On Thursday night, about an hour before going to bed, I decided to do an in-place upgrade to my VPS. I knew I had good backups of WordPress, the websites, etc. so I was not all that concerned. I have done many, many release upgrades of Ubuntu on-site I figured about 30-40 minutes for the upgrade. And indeed, after about 30 minutes the upgrade was finished and it was time to reboot.

I gave the VPS about a minute to restart and… nothing. I could not connect using SSH or using a web browser. I hop onto the VPS console and see a bootloop. Ok, I’ve seen this happen before. Jump into recovery mode and see what was broken and fix it. And it seems I really broke something as I could not even start the recovery console. I have done a couple of in-place release upgrades plus, of course, a bunch of tweaks over the last few years so I can see that causing issues.

Since I was tired I powered off the borked VPS and off to bed I went. I am finally old and wise enough (or maybe my ass was dragging…) to know being tired and being successful often does not work out.

The next day I installed a new instance of the VPS and began the recovery.

The good news:

• The WordPress recovery went fairly easily. The database backup was solid. I just had to remember to create the WordPress database account and grant it access to the WordPress database. Funny about that…
• The recovery of the other websites went fine.
• It gave me a chance to clean up some of the php code, old URLs that had existed for who knows how long, etc.
• Re-implementing Let’s Encrypt SSL was buttery smooth.
• But, more details below…

The bad news:

• I had meant to back up and copy offsite the WordPress content directory. I had part of it done – copy it it over to another directory – but I did not remember to set up the job to copy the contents offsite. Insert Homer Simpson Dunh! here. Then I remembered last month I was messing around with a full copy of my WordPress site locally. Awesome – I have a copy of the WordPress contents. Upload and fixed.
• I somehow did not have an anywhere near current copy of the other websites. That meant that other fixed that I had done over the years did not exist. However, it did mean I had the chance to update the php code.
• I also did not keep a copy of the ufw blocks for, shall we say, less than desirable ISPs. Backed up now and improved by cleanup.

Lesson learned: Make sure your backups are complete. Don’t assume anything.

After years of using desktop UPSes for my server and network gear, I finally bought a rack mount, data centre grade UPS. Over the years, APC was my go-to brand. However, after two of my three UPSes had their batteries die, that all changed especially after buying new batteries – even generic replacements – was approaching the cost of a new UPS.

In searching eBay for the replacement batteries I stumbled across a Canadian (useful due to shipping, import duties when buying from the US) that was selling rerfubished Lenovo RT1.5kVA UPSes with new batteries.

The RT1.5kVA also includes an SNMP network card so I no longer have to use USB connections and can easily have other devices like my pfSense firewall, Proxmox host and two Synology NASes “share” the UPS and not have to rely on some other device.

I now have just under 30 minutes runtime.

The last, just under 3-year-old APC UPS will now be reused for my ISP’s router and ONT. With 1500 watts available that is 5x more than the old 300 watt UPS that is getting rather old and probably needs a new battery, too.

Good bye 2023… Hello 2024! I wish everyone health, wealth and happiness over 2024.

I guess I’m getting a little nostalgic as I have become ye olde IT grey beard…

I thought I would put up some my “server closet” – actually, it was a closet – pictures and tech details from 2002.

The firewall ran SmoothWall 0.9.9. This was a highly modified and hardened VA-Linux kernel that provided firewall, Intruder Detection Systems (IDS) and proxy services.
Tech specs:

• Asus AP53 motherboard
• Pentium 166 MHz CPU
• 128 MB RAM
• One 340 MB IDE hard disk
• 24X E-IDE CD-ROM
• 3Com 3c905B network adaptor
• SMC Ultra network adaptor
• Matrox Millennium G200 PCI video card

The Windows 2000 Server was the file server, secondary internal DNS server and Active Directory Domain Controller. This system was used to run Internet Information Server 5 until it was hacked in the first Code Red attacks in July 2001. It also ran the RealProducer software using a Logitech Color USB WebCam. (I really wish it was an AlphaServer 4100!)
Tech specs:

• Soyo SY-7VBA 133 motherboard
• Celeron 700 MHz CPU
• 192 MB SD-RAM
• Two fixed 9 GB Ultra SCSI disk
• Adaptec AHA-2940U SCSI Controller
• 4/8 GB WangDAT 4mm DAT tape backup
• 48X E-IDE CD-ROM
• ATI Mach64 4 MB PCI video card
• 3Com EtherLink XL 10/100 PCI network card

And last, but not least, the HP NetServer LD Pro that ran Red Hat Linux 7.0. It used Apache with PHP and Perl as the web server. FTP services were handled by ProFTPD. The NetServer also provided primary internal dynamic DNS services with BIND 9. Live Web Cam services are provided by RealServer 8. Of course, it was constantly updated with security patches (after Code Red)!
Tech specs:

• Pentium Pro 180 MHz CPU
• 96 MB ECC RAM
• One fixed 9 GB Ultra SCSI disk
• Two hot swap 9 GB Ultra SCSI disks
• 24X SCSI CD-ROM
• Intel EtherExpress Pro/100 Plus network adaptor

Well, despite my misgivings over the price of Netgate’s rack mount for the 4100/6100 firewall series, I finally broke down and bought one. I still think it is far overpriced even with the nice blue anodised aluminium. I really wish that Netgate had created a proper chassis with integrated power supply for the 4100/6100 series. Not that I would be buying a replacement even if they did given the cost. It does look okay; but, in reality, a standard rack shelf is good enough.

Next, notice anything else new in the rack:

Hint: Is isn’t HPE but from IBM…

My son has made fun of me for not having “one of those pull out screens” for the rack. I actually have been looking for some time on eBay but the prices for any that will ship to Canada is like cra-cra even for those that are broken, unknown working, missing cables, missing rails, etc. However, I found one that was being thrown out (keyboard damaged, monitor unknown) but it has the rails, the cable management arm and both the rails and arm were only slightly bent (i.e., fixable).

The keyboard was the “classic” ThinkPad keyboard with the TrackPoint and touch pad. But it has PS/2 connectors (oh-so-retro) that neither the DL360 G8 or DL380 Gen9 has and I didn’t have a proper PS/2 to USB adaptor (most are just electrical pass-thru which does not work – for me at least).

The solution was to pick up a Lenovo ThinkPad Compact USB Keyboard with TrackPoint. Not having a track pad doesn’t bother me as my servers just run in text mode. I did have to modify the drawer as the compact USB keyboard is about 1 cm narrower and if bumped would drop the drawer. The drawer is designed with the original keyboard hanging through the drawer and held down with a big piece of Velcro (as is the monitor’s power supply – simple and effective). I solved this with a piece of 3/8 thick backer board and drilling some countersunk screw holes. But no Velco because I don’t have any and I don’t think it is needed.

I am now a “proper” nerd with a rack KVM console. No KVM switch because the two I still have are HDMI, not VGA – but my two servers (and old pfSense box) use VGA. There is some interference on the monitor (it is only 15″) and it uses some weird interface that the power and VGA connect. It is pretty thin and uses a standard VESA mount. I’ve looked around but these seem hard to get and I can live with the interference for the limited times I use it. I mostly use the iLO but there are cases when I need to be at the console.

The old DL360 G8 is only used for testing. I did manage to get the pfSense box working again after replacing the RAM, reseating the mSATA drive and re-installing pfSense. But, with a server, my backup UniFi Switch24 and a firewall, hmmm… Opportunity?

It has been just over 17 days with my new Netgate 6100. Shipping was sort-of a day late – I guess because couriers seem to have difficulty with delivery times to the easterly part of North America. I purchased the “base” model with 8GB RAM and 16GB of storage. I’m not worried about the storage since all the logs go to another syslog server anyway. And I’m not running a branch office or anything this is more than sufficient for my needs.

The migration from the old commodity-based router was fairly easy. I booted up the new router with my workstation connected to my planned “LAN” interface (not really as my configuration has a few VLANs, etc.) and the planned WAN interface into, well, the WAN, I allowed the 6100 to update to pfSense+ 23.01 (the most current). After looking up the network interface names from Netgate’s awesome documentation you only have to edit the XML file to replace the old network interface names with the new interface names. Then you restore the suitably modified backup file. Some additional time is needed to bring the additional services such as OpenVPN, pfBlockerNG, etc. to download and update.

The only problem I had was that despite adding the MAC address to my ISP’s router’s Advanced DMZ configuration inbound access was not working. After checking – and double checking – my configurations such as “Did I enter the correct MAC from the 6100?” I fell back on the old, default IT help desk recommendation… I rebooted and all was working again.

What I like about the 6100:

• Longer-term futureproofing: I now have 10GbE interfaces if I go above a 1GbE WAN connection and/or upgrade to 10GbE internally. The four “LAN” ports are actually 2.5GbE so more room there, too.
• pfSense is fully supported by Netgate on known hardware: Less worries about upgrades going wrong.
• Price: The price is essentially the same for a generic router with two 10GbE SFP+ ports, two copper/SFP shared ports plus four 2.5GbE ports – assuming you can actually find this configuration.

What I do not like:

• Having to buy a 1U adaptor. The price of US$107 is something I really do not like.