I was going through some old web backups (because of the last post) and found a picture (320×240 and I scaled it up a bit for this post – I think we had a Sony Mavica digital camera).
Here’s some of what I had put in place just before I left for a new job:
A SAN with Hewlett-Packard FC60/SC10 disks and Brocade Silkworm-based Fibre Channel interconnections
Hewlett-Packard NetServers
Open Storage Solutions servers and external disk arrays
Custom-built Pentium II / III rack-mount servers
Digital AlphaServer-based network support systems (internal/external DNSes)
Exabyte 220 tape library system
Yes, that is a Cisco PIX 1000 (or 500, I can’t actually remember but it used 3.5″ floppies for backups)
Edit: I just realized that there is an IBM workstation running RealProducer (lonely black tower in the middle of the picture) with a whitebox server to the left running RealServer. Those were the days 🙂
On Thursday night, about an hour before going to bed, I decided to do an in-place upgrade to my VPS. I knew I had good backups of WordPress, the websites, etc. so I was not all that concerned. I have done many, many release upgrades of Ubuntu on-site I figured about 30-40 minutes for the upgrade. And indeed, after about 30 minutes the upgrade was finished and it was time to reboot.
I gave the VPS about a minute to restart and… nothing. I could not connect using SSH or using a web browser. I hop onto the VPS console and see a bootloop. Ok, I’ve seen this happen before. Jump into recovery mode and see what was broken and fix it. And it seems I really broke something as I could not even start the recovery console. I have done a couple of in-place release upgrades plus, of course, a bunch of tweaks over the last few years so I can see that causing issues.
Since I was tired I powered off the borked VPS and off to bed I went. I am finally old and wise enough (or maybe my ass was dragging…) to know being tired and being successful often does not work out.
The next day I installed a new instance of the VPS and began the recovery.
The good news:
The WordPress recovery went fairly easily. The database backup was solid. I just had to remember to create the WordPress database account and grant it access to the WordPress database. Funny about that…
The recovery of the other websites went fine.
It gave me a chance to clean up some of the php code, old URLs that had existed for who knows how long, etc.
Re-implementing Let’s Encrypt SSL was buttery smooth.
But, more details below…
The bad news:
I had meant to back up and copy offsite the WordPress content directory. I had part of it done – copy it it over to another directory – but I did not remember to set up the job to copy the contents offsite. Insert Homer Simpson Dunh! here. Then I remembered last month I was messing around with a full copy of my WordPress site locally. Awesome – I have a copy of the WordPress contents. Upload and fixed.
I somehow did not have an anywhere near current copy of the other websites. That meant that other fixed that I had done over the years did not exist. However, it did mean I had the chance to update the php code.
I also did not keep a copy of the ufw blocks for, shall we say, less than desirable ISPs. Backed up now and improved by cleanup.
Lesson learned: Make sure your backups are complete. Don’t assume anything.
After years of using desktop UPSes for my server and network gear, I finally bought a rack mount, data centre grade UPS. Over the years, APC was my go-to brand. However, after two of my three UPSes had their batteries die, that all changed especially after buying new batteries – even generic replacements – was approaching the cost of a new UPS.
In searching eBay for the replacement batteries I stumbled across a Canadian (useful due to shipping, import duties when buying from the US) that was selling rerfubished Lenovo RT1.5kVA UPSes with new batteries.
The RT1.5kVA also includes an SNMP network card so I no longer have to use USB connections and can easily have other devices like my pfSense firewall, Proxmox host and two Synology NASes “share” the UPS and not have to rely on some other device.
I now have just under 30 minutes runtime.
The last, just under 3-year-old APC UPS will now be reused for my ISP’s router and ONT. With 1500 watts available that is 5x more than the old 300 watt UPS that is getting rather old and probably needs a new battery, too.
Good bye 2023… Hello 2024! I wish everyone health, wealth and happiness over 2024.
I guess I’m getting a little nostalgic as I have become ye olde IT grey beard…
I thought I would put up some my “server closet” – actually, it was a closet – pictures and tech details from 2002.
The firewall ran SmoothWall 0.9.9. This was a highly modified and hardened VA-Linux kernel that provided firewall, Intruder Detection Systems (IDS) and proxy services. Tech specs:
Asus AP53 motherboard
Pentium 166 MHz CPU
128 MB RAM
One 340 MB IDE hard disk
24X E-IDE CD-ROM
3Com 3c905B network adaptor
SMC Ultra network adaptor
Matrox Millennium G200 PCI video card
The Windows 2000 Server was the file server, secondary internal DNS server and Active Directory Domain Controller. This system was used to run Internet Information Server 5 until it was hacked in the first Code Red attacks in July 2001. It also ran the RealProducer software using a Logitech Color USB WebCam. (I really wish it was an AlphaServer 4100!) Tech specs:
Soyo SY-7VBA 133 motherboard
Celeron 700 MHz CPU
192 MB SD-RAM
Two fixed 9 GB Ultra SCSI disk
Adaptec AHA-2940U SCSI Controller
4/8 GB WangDAT 4mm DAT tape backup
48X E-IDE CD-ROM
ATI Mach64 4 MB PCI video card
3Com EtherLink XL 10/100 PCI network card
And last, but not least, the HP NetServer LD Pro that ran Red Hat Linux 7.0. It used Apache with PHP and Perl as the web server. FTP services were handled by ProFTPD. The NetServer also provided primary internal dynamic DNS services with BIND 9. Live Web Cam services are provided by RealServer 8. Of course, it was constantly updated with security patches (after Code Red)! Tech specs:
Every so often with pfSense I get “phantom” entries of pfSense’s hostname to incorrect VLAN gateway addresses. You can remove these incorrect entries – which appear in /etc/hosts but they will auto-magically reappear when Unbound is restarted.
Ever since moving to Pop!_OS 22.04 LTS wired networking have been very slow. I recalled seeing posts on this problem and the issue being the included driver for the RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller.
Performance was not only poor but downloads were much slower than uploads as evidenced by the Speedtest CLI package:
I’m not sure if this is an issue with Lenovo’s updating of the BIOS resetting the default boot back to Windows or if it is Windows itself. Since this has happened a couple of times, this is a reminder to myself on how to fix this:
Repair the bootloader – since even manually selecting Pop!_OS leads to a hang in UEFI
Well, despite my misgivings over the price of Netgate’s rack mount for the 4100/6100 firewall series, I finally broke down and bought one. I still think it is far overpriced even with the nice blue anodised aluminium. I really wish that Netgate had created a proper chassis with integrated power supply for the 4100/6100 series. Not that I would be buying a replacement even if they did given the cost. It does look okay; but, in reality, a standard rack shelf is good enough.
Next, notice anything else new in the rack:
Hint: Is isn’t HPE but from IBM…
My son has made fun of me for not having “one of those pull out screens” for the rack. I actually have been looking for some time on eBay but the prices for any that will ship to Canada is like cra-cra even for those that are broken, unknown working, missing cables, missing rails, etc. However, I found one that was being thrown out (keyboard damaged, monitor unknown) but it has the rails, the cable management arm and both the rails and arm were only slightly bent (i.e., fixable).
The keyboard was the “classic” ThinkPad keyboard with the TrackPoint and touch pad. But it has PS/2 connectors (oh-so-retro) that neither the DL360 G8 or DL380 Gen9 has and I didn’t have a proper PS/2 to USB adaptor (most are just electrical pass-thru which does not work – for me at least).
The solution was to pick up a Lenovo ThinkPad Compact USB Keyboard with TrackPoint. Not having a track pad doesn’t bother me as my servers just run in text mode. I did have to modify the drawer as the compact USB keyboard is about 1 cm narrower and if bumped would drop the drawer. The drawer is designed with the original keyboard hanging through the drawer and held down with a big piece of Velcro (as is the monitor’s power supply – simple and effective). I solved this with a piece of 3/8 thick backer board and drilling some countersunk screw holes. But no Velco because I don’t have any and I don’t think it is needed.
I am now a “proper” nerd with a rack KVM console. No KVM switch because the two I still have are HDMI, not VGA – but my two servers (and old pfSense box) use VGA. There is some interference on the monitor (it is only 15″) and it uses some weird interface that the power and VGA connect. It is pretty thin and uses a standard VESA mount. I’ve looked around but these seem hard to get and I can live with the interference for the limited times I use it. I mostly use the iLO but there are cases when I need to be at the console.
The old DL360 G8 is only used for testing. I did manage to get the pfSense box working again after replacing the RAM, reseating the mSATA drive and re-installing pfSense. But, with a server, my backup UniFi Switch24 and a firewall, hmmm… Opportunity?
Netgate 6100 Rear – Because the Front is Boring 🙂
It has been just over 17 days with my new Netgate 6100. Shipping was sort-of a day late – I guess because couriers seem to have difficulty with delivery times to the easterly part of North America. I purchased the “base” model with 8GB RAM and 16GB of storage. I’m not worried about the storage since all the logs go to another syslog server anyway. And I’m not running a branch office or anything this is more than sufficient for my needs.
The migration from the old commodity-based router was fairly easy. I booted up the new router with my workstation connected to my planned “LAN” interface (not really as my configuration has a few VLANs, etc.) and the planned WAN interface into, well, the WAN, I allowed the 6100 to update to pfSense+ 23.01 (the most current). After looking up the network interface names from Netgate’s awesome documentation you only have to edit the XML file to replace the old network interface names with the new interface names. Then you restore the suitably modified backup file. Some additional time is needed to bring the additional services such as OpenVPN, pfBlockerNG, etc. to download and update.
The only problem I had was that despite adding the MAC address to my ISP’s router’s Advanced DMZ configuration inbound access was not working. After checking – and double checking – my configurations such as “Did I enter the correct MAC from the 6100?” I fell back on the old, default IT help desk recommendation… I rebooted and all was working again.
What I like about the 6100:
Longer-term futureproofing: I now have 10GbE interfaces if I go above a 1GbE WAN connection and/or upgrade to 10GbE internally. The four “LAN” ports are actually 2.5GbE so more room there, too.
pfSense is fully supported by Netgate on known hardware: Less worries about upgrades going wrong.
Price: The price is essentially the same for a generic router with two 10GbE SFP+ ports, two copper/SFP shared ports plus four 2.5GbE ports – assuming you can actually find this configuration.
What I do not like:
Having to buy a 1U adaptor. The price of US$107 is something I really do not like.
I guess buying the not-so-cheap Alibaba-ish 1u rack router was not a totally great idea. Something failed and I think that it is network ports. All the network ports. All six of them.
Anyway, I have a Netgate 6100 ordered with one-day testing and express shipping by FedEd. One day shipping – I wonder how that will work out coming from the US – was only $7 more.
Once I get it in, I hope that simple editing of the backup file to line up the new interface names with the port assignments will put everything back to normal. The one thing I think I’m going to miss is the VGA output as it makes set up so easy. I wonder how the console connection will work out.
Not much to do now but wait. With pfSense down, there are no VLAN/subnets, DHCP, DNS, access any network resources like the NAS, etc.
