Machines
My lab consists of 5 machines: my production machine, my secondary, and 3 raspberry pis.
War Rig
My main production machine is a custom built PC that dual boots EndeavorOS Linux and Windows 11. I try to follow a naming convention for my machines, and since Mad Max: Fury Road is one of my favorite movies and for the general beefiness of the machine at the time of build, it’s dubbed The War Rig. The linked part’s list is a little out of date, with the notable upgrades being:
- I have two vertical 27” BenQ monitors on each side of the MSI monitor in an “H” configuration
- I upgraded to the Ergodox-EZ, and then was gifted the Voyager which became my daily driving keyboard. Using split keyboards has been an absolute game changer for me after the initial couple of weeks re-learning how to type.
- Because of my massive head and my habit of having 1 ear out of the cuff, stressing the band, I burn through headsets. I am currently using the Logitech G PRO X Wireless headset, which among other things has a metal band
The Windows 11 partition is only used to play a few steam games. If I’m not playing a video game, I don’t have any good reason to be on my windows partition, as most of my work and non-gaming related leisure is done in Linux.
There is no particular reason I am on EndeavorOS and not anything else. I’ve always liked Arch and Arch-based distros, and I stumbled on Endeavor one day and thought the defaults were great. Before EndeavorOS, I was on Garuda, and before Garuda I was on a base Arch install with KDE.
Citadel
Because I have amazing friends, I received a refurbished Dell Optiplex 7010 with an Intel i5 and 16 Gbs of ram as a Christmas present. I slapped EndeavorOS on it, and it’s been working as my secondary lab machine ever since.
Raspberry Pis
I have 3 raspberry pi 3s model B+ each running Hypriot, which is just Raspbian that has been optimized for docker. There isn’t much difference between the pis themselves. In the past, I’ve done some work around getting them to work as a kubernetes cluster, but that didn’t end up worth the effort to maintain.
The host names for the pis are all pirate ship names:
- Black Pearl (default hostname for hypriot)
- Wellerman
- Billy Of Tea
Software
What Machine hosts what?
There honestly hasn’t been much thought into which machine hosts which piece of software, and that hasn’t really come back to bite me yet. I have a pretty robust backup system that ensures that each of the important pieces of data gets backed up so that I can have some persistence during a disaster scenario, so there is no real concept of roles that each machine plays. That being said, there are a few considerations I make when deploying new software:
- Does the public docker image support the arm/v7 architecture?
- I’ve gotten around this in the past by using buildx to rebuild the images as arm/v7 to put them on one of the pis, but only if that is something I’ve had a use case for
- Do I have a reason to want to access this service if my main production machine is offline?
- Not very common, but still a consideration
War Rig Services
Bin
This is a pastebin service written in Rust that features server side syntax highlighting, image pasting, file drag and drop, and a neat cli tool. It doesn’t require a SQL server, and runs off of a statically linked binary. It certainly hits all of the wickets for a pastebin service.
CyberChef
This is the open source version of the https://gchq.github.io/CyberChef/ application.
CyberChef is similar to String-Is, but with some additional features like the “Magic” tool, which can intelligently decode a string. This comes in handy when practicing security work on HackTheBox or during my OSCP labs/exams. There are also many more tools at your disposal in cyberchef, and some addtional features like being able to take in files and sequence operations in a repeatable manner.
Dashy
I use Dashy as my HomeLab Dashboard. I can’t say enough good things about it, so I won’t.
Flatnotes
I experiment a lot with different ways to get notes into my Obsidian vault, and Flatnotes works well within my system since they save as just regular markdown files. Most of this experimentation is to fiddle with way to get around it taking a century for my Obsidian vault to load on mobile. All I had to do was map my docker container’s volume to a directory in my Obsidian vault, which is already synced to my devices with SyncThing.
Karakeep (Formerly Hoarder)
This is my default bookmarking service. It has some neat functionality around saving cached versions of web pages using monolith.
Jellyfin
Okay, its not what you think (though I’ve very much not opposed to what you might expect this to be used for). Honestly, I run my Jellyfin server because I enjoy watching my YouTube videos at 2.5 speed, I hate YouTube ads, and I don’t like logging all of my devices at home into my Google account to watch my subscriptions. Check out Feeds to see which YouTube channels I’m subscribed to.
Kutt
This is my default url shortener, which I use for tldr.cam links.
MeTube
While I have ytdl-sub for my subscriptions, I use metube to download one-off videos from folks I don’t subscribe to, which also goes to Jellyfin.
Noteshare.Space
This app is pretty old, but its solid for when I need to share an Obsidian note quickly and directly from the interface, without setting up any type of encryption.
Open WebUI
This is my default interface for AI. I’m able to hook in OpenAI and ollama into the same interface, and create tools and models for my needs. Since it interacts with OpenAI’s API, I get the benefits of access to newer models before they are available in the ChatGPT interface.
Photoprism
PhotoPrism is my photos management system, and is like an Open Source Google Photos. I am by no means a photographer, but I do like to keep my photos and have some ability to search them and find what I am looking for, and PhotoPrism makes that easy.
Portainer Agent
I utilize Portainer as the administrative interface for the majority of my services. To that end, I have an agent installed on all of the machines in my home lab that I intend to deploy containers on.
String Is
This is the open source version of the https://string.is web app.
Often enough, I find that I need to perform some conversion on text in order to make it useful. It could be something like decoding or encoding base64, turning json data into a csv, getting a timestamp into a different format, or checking a regex expression. String-is provides a swiss army knife for all of those operations.
SurrealDB
I use SurrealDB to play around with linked data. I don’t have many very formal uses for it yet, but I am storing some embeddings for some RAG applications I’m tinkering with.
Webhook Site
This is the open source version of the https://webhook.site/ web app.
This is useful for me when creating new automations that hook into various platforms using webhooks. Often, tools will provide the ability to post to web endpoints when events occur, either to generate notifications or perform some standard actions, however finding and understanding the schema of what gets sent to those endpoints is difficult. For example, at work we wanted to be able to create annotations in our Grafana instance whenever an Octopus deployment kicked off. Octopus allows event subscriptions to post to web endpoints, but understanding what needed to happen for that data to be useful for the Grafana API required understanding how the data gets posted out of Octopus, and to gain this understanding I just had the subscription send data to the webhook.site endpoint. From the UI, I was able to understand how the data get sent from octopus, and then understand what changes the automation needed to make to post the data to Grafana.
Whishper
I use whishper mostly to transcribe my dictations so that I can clean them up in Open WebUI (more about that in my How I Use AI post). Occasionally I’ll also use it to get subtitles for a YouTube video.
Citadel Services
BunkerWeb
I transitioned to BunkerWeb from Nginx Proxy Manager. I have 3 domains that route to my BunkerWeb instance, which then routes all of my traffic to the underlying applications from there. It creates, administers, and automatically renews Let’s Encrypt certs for all of my applications.
Monitoring Stack
What kind of SRE would I be if I didn’t have some monitoring in my lab. I only listed Grafana here, but the whole stack consists of Grafana, Loki for logs, Prometheus for metrics, and Alloy as an OTEL collector. This stack monitors most of everything else in my lab.
SearXNG
This is my default search engine. It aggregates results from the other major search engines for my queries, and I have it set up to produce JSON outputs for searches, which I then can use in Open WebUI as a built in web search integration.
Windmill
I honestly don’t know why I don’t hear about this app more often. Windmill is essentially my lab’s central source of automation. You can write scripts in basically any language you want, and have them execute on schedules or as webhooks like FaaS. I love this app so much, I introduced it to my day job and now we are on the cloud version. Really just an absolute rockstar of an application. I highly recommend checking out the docs:
Wellerman Services
AppRise API
Sending notifications to the right places and at the right times can be a challenge. Apprise is a very robust tool that creates integrations to numerous platforms, and the Apprise API turns that notification ability into an API Service.
Portainer
As previously mentioned, I utilize Portainer as the administrative interface for the majority of my services. This is the actual web service instance, which also administers the docker images on Wellerman.
WishThis
This is my default wishlist service, which I use to break out of the amazon monopoly.
Black Pearl Services
Memos
More hosting of Markdown files. I could probably get rid of this one in favor of Flatnotes and Noteshare.Space and this blog, but it doesn’t do me any harm having it running. It has a sync to Obsidian plugin, but it syncs using numeric IDs, and its only one directional, but sharing features are nice.
NTFY
I use this to programmatically send notifications to various devices including my phone, and also to perform actions on those devices when they receive certain actions. This is my primary route to be notified if my nightly backups for whatever reason aren’t working, and for status updates on administrative actions in my lab.
Portainer Agent
See previous comments. I’m going to skip this one on the following machines, but you can just expect it to be there.
Wakapi
I’m an absolute sucker for two things in this world:
- metrics
- open source api compatible versions of paid tools
This hits both of those. I’m lucky enough to not work in a place that tracks the data for coding time, but I do like to track it for myself. I mean, c’mon, this is cool:
Billy Of Tea Services
VaultWarden
VaultWarden is a BitWarden clone written in Rust, which implements all of the features of the BitWarden API including most of the paid services. I use YubiKeys for authentication to most things where I can, so I really wanted to have my web based password manager utilize my YubiKeys like my desktop password manager does (KeePassXC). Another shining example of a tool perfectly fitting a need.
Nginx Proxy Manager
Formerly the public face of my entire lab prior to my transition to BunkerWeb.
Fail2Ban
This is part of how I keep malicious actors out of the lab. This service ingests the logs from Nginx Proxy Manager, and if someone is acting fishy, it “bans” them. Banning in my case is adding a rule to CloudFlare to force the IP to jump through hoops for any of my sites before it gets access.