On Thursday night, about an hour before going to bed, I decided to do an in-place upgrade to my VPS. I knew I had good backups of WordPress, the websites, etc. so I was not all that concerned. I have done many, many release upgrades of Ubuntu on-site I figured about 30-40 minutes for the upgrade. And indeed, after about 30 minutes the upgrade was finished and it was time to reboot.
I gave the VPS about a minute to restart and… nothing. I could not connect using SSH or using a web browser. I hop onto the VPS console and see a bootloop. Ok, I’ve seen this happen before. Jump into recovery mode and see what was broken and fix it. And it seems I really broke something as I could not even start the recovery console. I have done a couple of in-place release upgrades plus, of course, a bunch of tweaks over the last few years so I can see that causing issues.
Since I was tired I powered off the borked VPS and off to bed I went. I am finally old and wise enough (or maybe my ass was dragging…) to know being tired and being successful often does not work out.
The next day I installed a new instance of the VPS and began the recovery.
The good news:
- The WordPress recovery went fairly easily. The database backup was solid. I just had to remember to create the WordPress database account and grant it access to the WordPress database. Funny about that…
- The recovery of the other websites went fine.
- It gave me a chance to clean up some of the php code, old URLs that had existed for who knows how long, etc.
- Re-implementing Let’s Encrypt SSL was buttery smooth.
- But, more details below…
The bad news:
- I had meant to back up and copy offsite the WordPress content directory. I had part of it done – copy it it over to another directory – but I did not remember to set up the job to copy the contents offsite. Insert Homer Simpson Dunh! here. Then I remembered last month I was messing around with a full copy of my WordPress site locally. Awesome – I have a copy of the WordPress contents. Upload and fixed.
- I somehow did not have an anywhere near current copy of the other websites. That meant that other fixed that I had done over the years did not exist. However, it did mean I had the chance to update the php code.
- I also did not keep a copy of the ufw blocks for, shall we say, less than desirable ISPs. Backed up now and improved by cleanup.
Lesson learned: Make sure your backups are complete. Don’t assume anything.
