On Sunday, I upgraded my Synology DS216+II to DSM 7.0. I followed the DSM 7.0 reviews by Robert (Robbie) Andrews on NAS Compares and made the plunge starting with the DS216 – my backup NAS. Things went well so on Wednesday evening my main NAS, the RS1221+ was upgraded. As my VMware ESXi 7.0 has its VMs on the RS1221 using iSCSI, I shut down the ESXi server because past experience has shown me that the remote datastores “disappear” when the NAS shuts down and won’t reappear until the ESXi box is rebooted. (More on that later, I may have found a workaround.)
Anyway, the RS1221 DSM 7.0 upgrade went well. For both the DS216 (which is technically underspeced as it only has 512 MB of RAM and 1 GB is recommended but it is on the Synology compatibility list) and the RS1221 the new interface is really snappy. Login is much, much faster. So far, so good.
After about a half-hour – enough time for the RS1221 to “settle in” – I powered on the ESXi server. And discovered that my iSCSI datastores were no longer there. First thing to check was the RS1221 to make sure that iSCSI was running and the LUNs with the VMs was still there. Ok, everything fine there. I could see the ESXi host connected on the RS1221.
Back to the ESXi server. Under storage I could see that the Software iSCSI adapter was there. So were the iSCSI targets. But no datastores… Ok, rescan. No dice, still no datastores. Time to reboot.
And… ESXi would not come up. It could not find the internal USB thumb boot drive. A moment of panic until I remembered I had (not recently but recent enough) backed up the ESXi configuration (twice – once to the DS216 and once to the RS1221). As I do not use vCenter, I had to backup the configuration from the ESXi shell. This will save a lot of time and effort instead of starting the configuration from scratch. That information can be found in the VMware Knowledge Base: How to back up ESXi host configuration. Time to grab a couple of new USB thumb drives (one for the installer, one as the destination) and make a new boot drive.
And… I could not get the new installation (not the installer) to boot. I could boot from the installer thumb drive and the installer could see the destination thumb drive and install ESXi. But there was no way that the server would boot ESXi after installation. Manually selecting the thumb drive didn’t work either. By now it was getting late and I had to work in the morning. Off to bed.
Thursday evening, and after a lot of trial and error I remembered that the three thumb drives I was using as targets have been used as FreeNAS/TrueNAS installers. Now, TrueNAS uses FreeBSD as its base and that seems to do some strange things to the signatures on the drive. Even with deleting the partitions and formatting the drive as FAT32 didn’t help. Weird. Finally, I also recalled that when I was first installing FreeNAS I had a similar problem with disk signatures. Solution: remove the disk signature on the target thumb drive and all was well – ESXi would boot. Since I had the backup of the ESXi box’s configuration, I applied the backup and the configuration returned.
But… the iSCSI datastores were still not there. It was getting late again and I had to work Friday. Off to bed for a not-so-restful night.
After a lot of Googling I found an article on VirtualizationHowto.com by Brandon Lee titled VMware ESXi 6.5 Can’t Add Existing iSCSI LUN (from back in December 2016 for ESXi 6.5!) that was most helpful. It is great for troubleshooting. Anyway, the high level solution (details in the article) was to:
esxcfg-volume -l
esxcfg-volume -m UUID
Magic – the datastores reappeared. (I had to do it twice – once for each of the iSCSI targets. I think the next time using a “-M” to make it permanent over reboots. I’ll get around to that.) This might also fix the need to reboot the ESXi box when I have to reboot the NAS or have to reboot the switch when I upgrade the firmware.
Now, on backups:
Synology has a free application called Active Backup for Business. You have to register when you install it, but it is still free. Active Backup for Business lets you backup your VMs even if they are live. If you do not have vCenter (I don’t), you need to enable ssh and ESXi Shell for Active Backup to work. (An aside: After enabling ssh and ESXi Shell the settings did not stay set after reboot. I think that was a warning that the original USB thumb drive was starting to fail.) You also need to enable Changed Block Tracking (CBT) on the ESXi host to reduce the transferred data size and time for backup because CBT backs up only the blocks that are changed since the previous backup. Synology (and VMware) has a nice article on how to do this in the KB article How to enable CBT manually for a virtual machine. Unfortunately, I have not found how to automatically backup the ESXi configuration yet. This is a great solution for Synology users.
The other thing is that I am, one could say, religious on backing up configurations. The pfSense configuration is backed up pre and post changes and upgrades as is the UniFi configuration. I back them up to three locations: the DS216, RS1221 and on my “work” laptop. I also keep previous firmware available so I can rollback if needed (or available – the DSM upgrade can’t be rolled back). Backups are a good thing!
