Importance of Backups and Wednesday Trials and Tribulations

On Sunday, I upgraded my Synology DS216+II to DSM 7.0. I followed the DSM 7.0 reviews by Robert (Robbie) Andrews on NAS Compares and made the plunge starting with the DS216 – my backup NAS. Things went well so on Wednesday evening my main NAS, the RS1221+ was upgraded. As my VMware ESXi 7.0 has its VMs on the RS1221 using iSCSI, I shut down the ESXi server because past experience has shown me that the remote datastores “disappear” when the NAS shuts down and won’t reappear until the ESXi box is rebooted. (More on that later, I may have found a workaround.)

Anyway, the RS1221 DSM 7.0 upgrade went well. For both the DS216 (which is technically underspeced as it only has 512 MB of RAM and 1 GB is recommended but it is on the Synology compatibility list) and the RS1221 the new interface is really snappy. Login is much, much faster. So far, so good.

After about a half-hour – enough time for the RS1221 to “settle in” – I powered on the ESXi server. And discovered that my iSCSI datastores were no longer there. First thing to check was the RS1221 to make sure that iSCSI was running and the LUNs with the VMs was still there. Ok, everything fine there. I could see the ESXi host connected on the RS1221.

Back to the ESXi server. Under storage I could see that the Software iSCSI adapter was there. So were the iSCSI targets. But no datastores… Ok, rescan. No dice, still no datastores. Time to reboot.

And… ESXi would not come up. It could not find the internal USB thumb boot drive. A moment of panic until I remembered I had (not recently but recent enough) backed up the ESXi configuration (twice – once to the DS216 and once to the RS1221). As I do not use vCenter, I had to backup the configuration from the ESXi shell. This will save a lot of time and effort instead of starting the configuration from scratch. That information can be found in the VMware Knowledge Base: How to back up ESXi host configuration. Time to grab a couple of new USB thumb drives (one for the installer, one as the destination) and make a new boot drive.

And… I could not get the new installation (not the installer) to boot. I could boot from the installer thumb drive and the installer could see the destination thumb drive and install ESXi. But there was no way that the server would boot ESXi after installation. Manually selecting the thumb drive didn’t work either. By now it was getting late and I had to work in the morning. Off to bed.

Thursday evening, and after a lot of trial and error I remembered that the three thumb drives I was using as targets have been used as FreeNAS/TrueNAS installers. Now, TrueNAS uses FreeBSD as its base and that seems to do some strange things to the signatures on the drive. Even with deleting the partitions and formatting the drive as FAT32 didn’t help. Weird. Finally, I also recalled that when I was first installing FreeNAS I had a similar problem with disk signatures. Solution: remove the disk signature on the target thumb drive and all was well – ESXi would boot. Since I had the backup of the ESXi box’s configuration, I applied the backup and the configuration returned.

But… the iSCSI datastores were still not there. It was getting late again and I had to work Friday. Off to bed for a not-so-restful night.

After a lot of Googling I found an article on by Brandon Lee titled VMware ESXi 6.5 Can’t Add Existing iSCSI LUN (from back in December 2016 for ESXi 6.5!) that was most helpful. It is great for troubleshooting. Anyway, the high level solution (details in the article) was to:

esxcfg-volume -l
esxcfg-volume -m UUID

Magic – the datastores reappeared. (I had to do it twice – once for each of the iSCSI targets. I think the next time using a “-M” to make it permanent over reboots. I’ll get around to that.) This might also fix the need to reboot the ESXi box when I have to reboot the NAS or have to reboot the switch when I upgrade the firmware.

Now, on backups:

Synology has a free application called Active Backup for Business. You have to register when you install it, but it is still free. Active Backup for Business lets you backup your VMs even if they are live. If you do not have vCenter (I don’t), you need to enable ssh and ESXi Shell for Active Backup to work. (An aside: After enabling ssh and ESXi Shell the settings did not stay set after reboot. I think that was a warning that the original USB thumb drive was starting to fail.) You also need to enable Changed Block Tracking (CBT) on the ESXi host to reduce the transferred data size and time for backup because CBT backs up only the blocks that are changed since the previous backup. Synology (and VMware) has a nice article on how to do this in the KB article How to enable CBT manually for a virtual machine. Unfortunately, I have not found how to automatically backup the ESXi configuration yet. This is a great solution for Synology users.

The other thing is that I am, one could say, religious on backing up configurations. The pfSense configuration is backed up pre and post changes and upgrades as is the UniFi configuration. I back them up to three locations: the DS216, RS1221 and on my “work” laptop. I also keep previous firmware available so I can rollback if needed (or available – the DSM upgrade can’t be rolled back). Backups are a good thing!

About Mike Pelley

Let’s see… A little about me… I’ve been around information technology since 1983 with computers such as DEC Rainbows (weird machine – the standard DOS couldn’t format its own floppy disks – remember them? – and I had to format them on a friend’s IBM PC) to Radio Shack TRS-80 to Apple ][e and Apple //c in the beginning. I have programmed in 8-bit assembly language on 6502, FORTRAN and COBOL on IBM System/370 (and I still hate JCL), VAX BASIC and COBOL (and a weird and massive WordPerfect 4.0 macro) on DEC VMS (Alpha), C/C++ on Digital Unix (ALPHA), and C/C++, Perl (it may be powerful but I still hate it), PHP on Linux (Red Hat, Centos, Ubuntu, etc.). I have work with databases such as Digital RDB (later to become Oracle RDB), Oracle DBMS, Microsoft SQL Server, MySQL and PostgreSQL on VAX, Alpha, Sun and Intel. Check out my professional profile and connect with me on LinkedIn. See I still think that Digital created some of the best ideas in the world: VAX clustering, DSSI disks (forerunner to SCSI) and the Alpha processor (first commercial 64-bit processor – Red Hat screamed on an Alpha!). DEC just could not seem to be able to give air conditioners away to someone lost in the Sahara Desert! VMware is one of the best ways to get the most out of an x64 server. And I have tried Oracle VM, Virtual Box and Microsoft Virtual Server. Outside of that I am a huge military history buff starting in the early 20th century. I love Ford Mustangs (my ’87 Mustang GT was awesome) and if I had the money I would have a Porsche 928S4. If I had a lot of money I would have a Porsche 911 Turbo. I also play too much AmrA 3 Exile mod. Over 5,000+ hours... I have a wonderful son, Cameron. I have a long suffering (Do you really need all that computer junk?) wife, Paula. I live in Paradise, Newfoundland and Labrador.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.