Kaizen #4: Overhauled homelab - Tim Hårek

I started this “blog-series” to force myself to write more about my set up, and my continuous iterations. But it didn’t happen… until today! This post has been a long time coming, and I have kept the ideas in my head and notes.I will walk through everything I did, in order:Previous setupNetwork situationWhat I self-hostedNetworkingFiber adventureWiringWiring the garageThe new serverThe new setupDebianDokployMediaJellyfinImmichFilesMonitoringMiscellaneousqBittorrentLinkdingMesh VPNFinal resultWhat’s remainingPrevious setupTwo years ago, I revamped my homelab. I had just bought a new machine to use as my main home server, coming from a 9U server rack with a bunch of machines tangled together. Back then I finally sat down to figure out how I want things. And it has worked great… until it hasn’t.The new machine back then, a Zimaboard, was set up with NixOS, which seems excellent and worked perfectly until I wanted to do a basic cronjob for backups. I never figured out how to do that, so I basically did my backups manually whenever I had the time. I use Restic, which is awesome.Network situationWe have a 750/750 Mbps fiber-connection with Bergen Fiber (Altibox), which is more than enough for our needs. It’s a very reliable connection, never slow, absolutely no complaints about the connection.Earlier this year we moved to a house, where we own the whole thing, it’s amazing! But the network situation wasn’t great. Not a single Ethernet port in the whole house, except where the fiber and modem were located. And guess where that’s located!? Beside the door to the balcony, where the previous owner used to have their TV (we have made a separate TV room). One thing to note, there was a bunch of phone lines throughout the house, and we’ll get to those later.My previous setup, underneath the cat's feeding station.As you can see, not an ideal placement, especially after we found out that it was the perfect location for the cat to have his food, away from the dog.

Another thing that I particularly didn’t like about the Zimaboard setup I had done, was that I chose to install NixOS directly on the onboard flash storage. Whenever I tried to upgrade the NixOS version, I ran into issues with having enough space. I had to do a bunch of clean ups and micro adjustments every fucking time to get it to work.My networking situation was fine after moving into the new house, but I was unable to get WiFi in the garage. The coverage seemed fine, but it wasn’t ideal outside or in the TV room. I don’t think having the router on the floor was the best spot, but I didn’t want to mount the thing on the wall of the living room either.All the WiFi came from an ASUS RT-AX92U router, which worked pretty well, especially on the second floor where it was located. But it was a whole different story on the first floor, and outside.What I self-hostedFor all my ripped media, TV shows and movies, I used Jellyfin, which is so good, it just works. I also used Jellyfin for audiobooks with a great iOS app, plappa. And for all my Linux ISOs, I used qBittorrent. It runs great as a headless torrenting client, and there’s also a great iOS app for it, qRemote.I also had a bunch of small Docker containers, for Foundry and some personal projects.And I used to self-host Nextcloud, but it became cumbersome and just a hassle to maintain. It was bloated for my use-case.NetworkingI bought Ubiquiti UniFi gear, specifically:UCG UltraU7 Lite access points x3PoE-injectorsI also bought a fresh roll of Ethernet cable, 500 meters, and tools to make my own cables.UniFi has a lot of respect from homelabbers, and for good reason. It’s not (F)OSS, but it’s excellent hardware with no subscription and no account required to use it.The easy part: unboxing all the new gear.Buying all the equipment was the easy part, putting it all together, oh man, it was a totally different ordeal.

Fiber adventureOne of the first things I thought about when we bought the house was to move the fiber inlet and modem to the attic, so that it could be out of the way and not be an eyesore. I politely asked my local ISP, Bergen Fiber, if they could do it for me, or if they could provide me with a 5 meter fiber-cable SC/UPC. They said yes to providing me with a cable, and when I followed them up via email asking what I needed to do for them to send me the cable, they replied via SMS that they couldn’t 🙃. They could send me a 2 meter cable though. So I asked around, and spent some time looking for the specific cable online. Most of what I found was the wrong length or way too expensive for a short cable. I managed to find one, from a B2B company that is 5 minutes away from where I live. I asked a plumber-buddy of mine that shops there if he could pick up the cable, and he did for 32 kr (~ $3).WiringBesides the fiber cable looking adventure, I had to wire up the whole house, especially for my new access points. I needed cables going from the attic to the living room, downstairs hallway, and to the garage in a separate building. In addition, I wanted Ethernet in my home office and TV room. How hard can it be?I asked a buddy of mine to help out figuring out the locations for all the things, and helping me pull cables. I told him it would take a few hours. We spent six hours, and he had to go home, I continued for three more days after that. I thought it was going to be easy…We started by moving the fiber and modem, which was easy, made a 42 mm hole in the wall from the attic and pushed a conduit down the wall to the living room, then we pulled the fiber. Worked like a charm!Then we made a hole in the middle of the ceiling in the living room for the PoE access point, that also was easy.But then came the hardest part, going from the attic, down two floors, without any existing conduit.

Like I mentioned, there were phone cables throughout the house, but none of them were wired up, and the conduit didn’t lead anywhere near the attic. We spent the majority of the initial day pulling on the phone cables to figure out where they went and how we could get the wires from the attic. I ended up making a hole above a phone connection box in the living room, so that I could pull conduit from the attic down to the box. Because this box lead to the first floor, in the entrance.Finding exactly where the wall lined up, from up in the attic.Making holes and fishing cable down through the wall.The hole I made above the old phone connection box in the living room.Pulling conduit from the attic down to the phone box.And the actual cable-making? What a fucking mess that was. I knew there was an order to do things in, but the ones I bought, had no documentation, so I just relied on the internet to figure out the order. It took me two hours to get a working cable, because I didn’t notice there was another transparent plastic bag within the transparent plastic bag with “guides” for keeping the twisted pairs in order. I only realized that when I read the bad reviews. I used a super simple cable tester to check that I made the cables right, it was a super worthwhile thing to buy, it would have been a headache without one.Wiring the garageMy original plan was always to wire Ethernet to the garage, but I didn’t know how to do it. I knew there was conduit going from the house, but I had no idea if there was a room for an Ethernet-cable. Luckily, I found a cut telephone-wire in the garage coming from a yellow conduit, which was the same color as the conduit in the entrance where I had already wired the first floor! I was certain it was the same conduit, so I started pulling the cable to see if it moved the cable from the house, but nothing happened. I did all sorts of shenanigans to make the cable move, but it didn’t flinch, not one bit. I ended up ruining the insulation of the wire, that was how tight it was stuck.The old telephone wire, stuck solid in the garage conduit.

By pure coincidence, I found the old coax and telephone box still attached to the house outside. I opened it up, I loosened all the wires from the contraption, and followed the cables down the conduit on the outside of the house. Lucky for me, the ground outside is gravel, so I just started to dig. And what do you know, I found the fucking conduit, in fact, I found three! One for the house, one for the garage, and one for what I think is the previous ISP/telephone company. I just cut them all, and pulled them out. And now the telephone-wire in the garage came loose like nothing.The old coax and telephone box I found still attached outside.I pulled the cable from the house through the conduit and into the garage’s conduit, and voila! We have Ethernet in the garage!Digging along the foundation, and finding three buried conduits.The new serverI managed to get my hands on a huge HPE DL380 Gen10 server as decommissioned hardware. It has two dedicated Intel Xeon CPUs and 288 GB RAM. I got 16x 600 GB enterprise 10k RPM HDDs, I haven’t put in the last 8, but it’s on the to-do list! The 8 that are put in are in RAID6, so I have two that can die before I have to switch them out. _,met$$$$$gg. tim@hailmary ,g$$$$$$$$$$$$$$$P. ------------- ,g$$P"" """Y$$.". OS: Debian GNU/Linux 13 (trixie) x86_64 ,$$P' `$$$. Host: ProLiant DL380 Gen10 ',$$P ,ggs. `$$b: Kernel: Linux 6.12.94+deb13-amd64 `d$$' ,$P"' . $$$ Uptime: 3 days, 2 hours, 42 mins $$P d$' , $$P Packages: 406 (dpkg) $$: $$.

- ,d$$' Shell: bash 5.2.37 $$; Y$b._ _,d$P' Display (VGA-1): 1024x768 [External] Y$$. `.`"Y$$$$P"' Terminal: /dev/pts/0 `$$b "-.__ CPU: 2 x Intel(R) Xeon(R) Bronze 3204 (12) @ 1.90 GHz `Y$$b GPU: Matrox Electronics Systems Ltd. MGA G200eH3 `Y$$. Memory: 8.58 GiB / 267.31 GiB (3%) `$$b. Swap: 22.89 MiB / 127.66 GiB (0%) `Y$$b. Disk (/): 955.36 GiB / 3.10 TiB (30%) - ext4 `"Y$b._ Local IP (eno4): 192.168.1.23/24 `"""" Locale: en_US.UTF-8 My initial plan was to have this server in the attic beside the fiber/modem and router, but after firing the machine up, I quickly realized that I would be able to hear the machine through the ceiling. Good thing I have plenty of room in the garage’s loft.I never measured the power draw before and after, and honestly it’s not a big concern for me, electricity is fairly cheap here in Norway.The machine is named hailmary, after the ship in Project Hail Mary. If you look at the neofetch above, you’ll notice it has a butt load of RAM, 288 GB, and I’m barely using any of it. I still have to figure out what to do with it all. I’m thinking about getting a GPU for the server, for running local LLMs on it.The ultimate server setup: straight on the garage loft floor.