Mike Pelley's Musings

Back in August I posted on my latest updates to the home lab. At that time, I had added a new HP DL360p Gen8 to replace the old G7 for ESXi, added a UniFi Flex Mini for the office and replaced some of the old CAT5e “thick” cable with the Monoprice Cat6A SlimRun patch cables. I was planning some additional upgrades/replacement when my father fell ill in September and then passed away in October. Since dad died my sister and I have been quite consumed with being executors of his estate.

Since then, I have been able to make a few upgrades and replacements:

• I replaced the second UniFi UAP-AC-Lite upstairs with a Unifi UAP-FlexHD. That is a nice AP even though it looks like someone painted a Red Bull can white 🙂 That said, it is very unobtrusive and works really, really well.
• The UAP-AC-Lite was moved to my shed. The placement of the APs in the house coupled with the number of walls (some concrete) and the metal garage door led to terrible WiFi performance.
• I really disliked the “wall warts” for the UniFi US-8 switches and the PoE injectors took up space in the rack (they were mounted on a board at the top of the router). Since I now had six devices that could be powered via PoE, I went with the UniFi US-8-150W switch.
• Since I was at it – and wanted to free up some ports on the UniFi US-24-G1 – I added a couple of UniFi Fiber Multi-Mode SFP 1Gbit/s modules with a 10Gtek LC to LC OM3 10Gb/Gigabit Multi-Mode Fiber Optic cable. Thinking back, I may have been better off (no real cost difference) with using a DAC cable instead. Maybe later, but I think that it “kinda kool” to have fibre in my rack even if it is pointless…
• I added a Detroit Packing CAT6a patch panel. This has female-to-female keystone jacks since I didn’t want to punch down/crimp cables and I wanted to keep using the SlimRun patch cables.
• I continued to replace any of the thick cables where possible with the SlimRun cables.
• Cable labels – where needed I added labels especially from the ESXi boxes and the TrueNAS box.
• Wire Management: I have some small zip ties used for the ESXi/TrueNAS (more below) cable bundles. However, for other bundling, I have used hook-and-loop (a/k/a Velcro) wraps.
• Speaking of which:
  • FreeNAS is dead, long live TrueNAS. iXsystems has merged FreeNAS with TrueNAS creating TrueNAS Core. A nice few updates, etc. As of this post, I’m running TrueNAS-12.0-U3.
  • pfSense is now running 2.5.1-RELEASE; there was that awful mess (to be very kind) with 2.5.0 and WireGuard if 2.0…
• I added the old door Information Systems sign (when our group changed names in my first job over 25 years ago!). Why? Why not?

So, what’s next? I’m not quite sure. I was thinking about replacing the the Supermicro TrueNAS server with a HP DL380 Gen8. I like the HP BMC as it uses a HTML5 remote console instead of Java like the Supermicro. But, given that work-from-home/school-from-home enviornment that is likely to continue for some time, I’m thinking about redundancy. This is especially the case for the pfSense box. I can get away for a period of time replacing the Switch 24 (not without pain) but pfSense is another matter. I thinking about buying anothe box just like the one I have. If I buy a UniFi Switch 24 G2 and another pfSense box I’ll have perfect components for my home lab – moving PROD away from DEV 🙂

Well, I’m almost at the end of the server/network upgrade for now. Since the last update, here’s where we are from the last update:

1. I rebuilt the network configuration (about 85% of it).
   1. I had a weird problem with DHCP requests. The DHCP address a client requested (except for the two WiFi VLANs/subnets) would not get an address assigned to the subnet. It seems through the evolution of my network I had assigned a subnet to a physical interface and a VLAN on the pfSense firewall. Since I’m not a network engineer by any sense of the term (with my day job I feel like I’m 1 mm deep and 1 km wide) so there may be some reason why you would do this. Anyway, I removed the VLAN and made that subnet my default “core” network.
   2. I think that, for some reason (maybe the issue above), the two UniFi Switch 8s (non-PoE) were funky with DHCP requests, VLAN assignments and connections between the switches (e.g., Switch 8 in my son’s gaming room to the Switch 8 in the TV room to the Switch 24 in the rack). Google University and the forums didn’t give me much help outside of the the suggestion that the configurations for the Switch 8s might be corrupt. I reset both the Switch 8s to factory configuration and deleted them from the UniFi controller. Once I did that I re-adopted the two Switch 8s into the UniFi controller and reconfigured the VLANs on the ports everything works just fine. Plus, the Switch 8 in my TV room could power the Switch 8 in the gaming room. (The Switch 8’s can be powered by PoE and one port can provide PoE. That was a nice bonus.)
   3. I added a UniFi Switch Flex Mini for the desk in the office. I can put VLANs on that small switch for testing, etc.
   4. The move to the Monoprice Cat6A SlimRun patch cables is almost completed in the rack. I only need to buy another pack of 1 foot (or 2 foot) patch cables. I ordered some 6 inch patch cables and only one was long enough to use. Dunh! I have 10 foot cables for the servers: yellow for FreeNAS (eventually to be TrueNAS), red for the HP DL360 G7 and orange for the DL360p Gen8 (more about that below). I bundled the servers patch cables into umbilicals. Connections to non-network/non-server devices are purple SlimRun patch cords (some more still need to replace a few older runs). White will be used in the rack for in-rack networking (e.g., those 6 inch ones).
2. I bought an HP DL360p Gen8 with 2 x Xeon E5-2650 2.0Ghz 8-Core CPUs, 128 GB of HP SmartMemory (8 x 16GB) PC3-12800R (DDR3-1600) Registered ECC Memory, 4 x HP Enterprise 300 GB 6G SAS 15K SFF Hot Plug Hard Drives, HP Embedded Smart Array P420i/1GB FBWC RAID Controller and 2 HP power supplies. I also have iLO 4 Enhanced which allows for HTML5 remote console. The DL360 G7 only supports, now anyway, remote console under Windows with the HP iLO Integrated Remote Console application. I messed around a little bit with trying to get the Remote Console application to work under Mint with wine, but couldn’t get it to work. I didn’t mess around with it for too long, but that is a sign that remote management is on its way out. (The Supermicro Java iKVM app only works with Firefox and IcedTea – for now…).
3. I installed VMware ESXi 7.0 on the Gen8 and moved the VMs over. The VMs are stored on the FreeNAS server using NFS. I’d advise using NFS rather than iSCSI as you can easily share the storage between endpoints (ESXi or mounting them from a workstation). The G7 is running ESXi 6.7 – a fresh install from the ESXi 6.0 install. The G7 is now used for testing and experimentation. When I’m not doing anything I turn it off.
4. I had one 300 GB SATA drive in the FreeNAS box but remembered I had two more 320 GB SATA drives in the old QNAP 2-bay NAS. I deleted the 300 GB pool (the first one that added to the FreeNAS box when I was first setting it up) and added the two 320 GB drives (all 8 bays are now filled) and created a 600 GB “small and slow” RAID-5 pool for the G7 to run test VMs on. Given that it is only 600 GB it might actually force me to delete old VMs. 🙂
5. I added another UPS, an APC BackUPS 1500. The older BackUPS XS 1300 was a little taxed with everything on it. The server power supplies are split between the two UPSes and the network gear is on the 1500.
6. Finally, for aesthetics I added 1U filler plates. After looking at the price of the metal filler plates on Amazon and eBay (what $20/each!?!?!) I made some out of some backing board I had left over from the homemade rack. I learned that even with a circular saw I couldn’t cut a good straight line so I borrowed my neighbor’s table saw (Thanks, Phil!) and re-cut them. I still need to paint them black (at some time – I hate painting as I always make a mess…). Here’s what it looks like now (The two chassis on the bottom might be used for some additional SAS storage in the future but I mounted them to get them out of the way. They don’t look too bad there.):

If you look at the previous posts, you can see the old – in some cases, really old – network cables that I was using in the rack. Some of these were even older than my son! They were a mix of various lengths, colours, and ratings. Some were CAT5, some were CAT5A and a couple were even STP (Sheilded Twisted Pair) CAT5. Those STP cables are really stiff. I have no idea where I accumulated them over time.

I was looking to replace them not only because of their age and ratings but also because many of them were too short. It may seem strange that to make your wiring neat that being too short would be a problem. The issue is that you cannot route and arrange short lengths neatly or efficiently. As well, given that both servers are now using the appropriate sliding rails, you may not have enough length to pull a server out.

After looking around, I found some SlimRun CAT6A cables from Monoprice. These cables are really thin! Four of the SlimRun cables take up only slightly more space than the old CAT5 cables as you can see with the two types of cables next to each other.

These are great cables because they also have hoods so the clips don’t snag when pulling them through the rest of the cables. The hood also allow you to easily push the clip down when removing a cable.

Now, this is much better. It is easy to route the cables through the loom and you can now see the switch’s indicator lights.

I bought three colours (3 sets of 5): red for the ESX server (that’s the DL360 G7 under the switch); yellow for the FreeNAS server (the Supermicro server under the DL360) and purple for everything else. (Why purple? I donno…)

All that is left is to replace the two keystone wall plate with a six keystone version to make get rid of the wires still hanging down from the ceiling.

In my previous posts, I noted that my homebuilt rack worked well execpt for two issues: (1) I was silly and went with telco threaded rails and (2) I neglected to add the 1/8th inch (3.2mm) thickness of each rail in the width between the 2×4 uprights; that is 1/4 inch (6.5 cm) too narrow.

That, of course, caused two problems. First, the rails for my Supermicro server are made for squre holes and could not be converted to use threaded holes. My DL360 does have adaptable rails whereby the studs could be removed and, with care, could have screws used – I had to line up the screw with the threaded rail holes as it was thread-to-thread between the vertial rails and DL360 rails. That meant that the Supermicro had to rest on top of the DL360. Not the best if I had to slide out either server.

The second problem was that even if I could have mounted the Supermicro rails, there was only a paper-thin gap between the Supermicro chassis and the rack’s rails. Grrr… I could have fixed that problem by taking everything out, getting the router out (after I found a router bit that was long enough for the 1.55 inch (4 cm) depth of the rail) and countersink the rails. I could not take the rack apart because I did such a good job with glue putting the rack together. I would also have to buy four new vertical rails. I couldn’t find 20U square hole rails anywhere and what I could find was 14U at over CDN$75.00 per pair. I would have to buy four (4!) pairs and cut two pairs down. That would be over CND$300.00 just for the rails. I could have search more but anyway…

I found on Amazon when I looking for the square-hole rails a rack from Raising Electronics similar to this one by Sunlight Machinery (they don’t have the 600mm depth rack listed right now) for just over CDN$200. IT was 27U, instead of 20U, but vertical space wasn’t a problem.

The Amazon comments noted that you need to chase out the threads before assembly because the power coating – which seems really good – was in the threads. I only had two of what seems like a hundred bolts/screws you need for assemby; so that wasn’t a big issue. The comments also noted that there alignment was not always perfect. Again, I only had two holes that didn’t line up perfectly; I think that they were the same ones that had a little powder coat in them. The biggest problem was the so-called assembly instructions. They are not assembly instructions; they are the exploded engineering view. And the diagrams are tiny. Did I say they were terrible?

Anyway, it took about an hour, with the help of my son, to assemble. It really is a two-person job.

It starts out like this:

Once assembled it looks like this: You can see the Supermicro rails installed for testing. The shelf and the wire management are not included:

The next day, with the help of my son we stripped out all the equipment from my homemade rack. (I think he was eager to help when I mentioned all Interent access was going to be down until I had everything reassembled :-)). After about three hours of checking the mapping of server ports to switch ports it not looks like this. The one issue I has was that the I forgot that igb0 is on the bottom-left of the Supermicro NICs. Once that was done, everything was complete.

I still have to work on the vertical wire management which likely means that I need to get new network cables. That isn’t a bad thing because the new ones will be CAT6 and have some appropriate boots to keep the plugs from snagging on the other cables (and breaking off). The Supermicro bottom NICs have next to no room to apply pressure to depress a boot to remove the cable.

My netwok upgrade is not complete. The last two pieces were to replace the 1GigE switch in the TV room and the 10/100 Mbit/s switch in my son’s gaming room.

A few months ago, my son questioned why his gaming PC (i7-4970K, 16 GB RAM, GTX 1070 superclocked, GigE NIC) could only get about 97 Mbit/s. He knew that we have at least 500 Mbit/s bidirectionally. I finally fessed up and let him know that the switch that his PC (and the PS3, PS4 xbox 360) was an old D-Link DSS-8+. I mean really old; so old that I can’t even remember buying it. Actually, I think that it was a Christmas present.

The TV room had an more modern D-Link DSG-108G GigE switch. Thus, the old 10/100 Mbit/s switch connected to the GigE switch and then into the the UniFi Switch 24 because I had no easy way of running cable from the gaming room to the office (drywall instead of drop ceilings).

The solution, of course was more Ubiquiti UniFi gear. In this case, two UniFi US-8 non-PoE 8-port switches. I don’t need the full PoE swtiches (e.g. the US-8 60 watt or 150 watt switches). The US-8 does allow for PoE 48V Passive input to power it, and the US-8 will provide 48V Passive output using the included power adapter, it wouldn’t work over the run between the TV room and the gaming room. I don’t think that it is really supposed to work, anyway. It does work over a 30 cm patch cord, though.

Anyway, except for that old DSG-108G on my desk (two laptops and printer – I don’t want to backhaul 3 long cables from the desk to the UniFi Switch 24) and the 5-port GigE unmanged switch for the physical edge DMZ, everything is now UniFi.

The physical networking is now done. I still have some VLAN cleanup to do – move the IoIT devices onto the distrusted network (“untrusted” ain’t a real word, apparently). Everything is done now except a new cable run and a new, real rack. My birthday is coming up in a couple of months…

I received my second UniFi AP Lite yesterday. The plan was to have two APs in the house – one on each floor – from the beginning. The first AP was mounted in the basement ceiling which provided good coverage for the basement but parts of the main floor and outside the house were spotty. This was likely because of the floor in between the basement and the main floor. Outside was likely due to the concrete walls 🙂

The implementation was almost seamless – except for stupidity on my part. When I configured the port for the new AP, I added the management VLAN which allowed adoption and the firmware upgrade. However, clients could connect but would not maintain connectivity. The UniFi controller showed a boatload of DHCP failures. What was going on?

Well, another case of rushing things. If you have your WiFi network on different VLANs those VLANs also need to be applied to the switch port as well. Otherwise, when a client connects to a WiFi network, which is different from the management network, the switch ports sees the DHCP request and says “I don’t know about network x! Go away!”. Anyway, once I added the correct profile (I love the network profiles that Ubuquity has in their SDN) with all the right VLANs things go as smooth as silk.

So, the network upgrade is now 99% complete. WiFi connectivity is great across the basement and main floor. I can actually now get a reliable, albeit a little weak, WiFi connection in my shed. I still have to replace some cable runs with Cat5e or Cat6 and desperately need to clean up the rat’s nest in the server rack. Maybe Santy Claus will help me out there.