Why NixOS stopped my distro hopping

I've been using NixOS for a while now, on my laptop and on the servers I self-host my stuff on. This post explains how I use it and why I think it's the best setup I've had so far.

Before NixOS I used Arch. I liked it, but I got tired of my system slowly drifting: packages I'd installed once and forgotten, config files I'd edited by hand, leftovers from things I'd tried. After a while it was full of garbage and I couldn't tell anymore what I had changed. With NixOS that doesn't happen, so I haven't had a reason to switch since.

My setup

I have three machines:

• balrog, my laptop, running Niri with DankMaterialShell.
• cyboogie, a server that hosts most of what I use day to day: photos with Immich, passwords with Vaultwarden, mail, music, documents with Paperless, my own Git forge, and a lot more. Around fifty services in total.
• gila, a small VPS that monitors cyboogie and sends me a notification when something is down.

All three are configured from the same Git repository.

Configuring everything in one place

On most distros you install the system, then install packages and edit config files as you go. After a few years you end up with a machine where nobody really knows what was changed and why.

On NixOS you don't do that. The whole system is described in Nix files, and you apply them with nixos-rebuild switch. If I want Vaultwarden on my server, I add this to the config:

services.vaultwarden = {
  enable = true;
  config.ROCKET_PORT = 8222;
};

and rebuild. If I remove it, it's gone from the system on the next rebuild. Nothing is left behind.

Since the config is in Git, I also get the history of every change, and I can look up how anything is configured with a simple search in the repo, instead of looking around in /etc.

Rolling back

Each rebuild creates a new "generation", and the previous ones are kept. If an update breaks something, I can choose an older generation in the boot menu, or run nixos-rebuild switch --rollback, and I'm back to a working system.

Deploying from my laptop

I don't need to log into the servers to update them. Since all the configs are in the same repo, I can build any machine's system from my laptop and send it over SSH:

nixos-rebuild switch --flake .#gila --target-host gila --sudo

This is great for gila. It's a small VPS that would take ages to compile anything, so my laptop does the building and gila just receives the result and switches to it.

Erasing the system on every boot

Because the system is rebuilt from the config anyway, you can go one step further. On my laptop, / and /home are reset to an empty state every time it boots. Only the folders I list in my config are kept, using the impermanence module:

home.directories = [
  "Documents"
  "Downloads"
  ".ssh"
  ".local/share/fish"
];

I like this a lot. Programs can't leave files around my home folder forever, and the list of persisted folders is also the list of what I need to back up.

Running a server with fifty services

This is where NixOS helps me the most. Maintaining that many services alone would be a lot of work, but most of it is automated now:

• Every input of my config is pinned in flake.lock, and every container image is pinned to a specific version and digest, so nothing changes unless I update it.
• Every week, a job on my Forgejo instance updates flake.lock and opens a pull request. Renovate does the same for container images, and merges minor updates automatically.
• On every pull request, CI builds the configuration of all three machines. If an update breaks something, I see it there before it reaches the server.
• The builds go into my own binary cache, served by Harmonia, so my laptop downloads what CI already built instead of compiling it again.

Secrets

Passwords and API keys are in the same repository, encrypted with sops-nix. Each machine decrypts them with its own SSH host key, and only gets the secrets its config uses. When I set up a machine, I don't have to copy secret files around by hand.

The downsides

NixOS isn't easy to learn. You need to learn the Nix language, which is unusual, and the documentation is spread between the manual, the wiki, forum posts and other people's configs. Error messages can be hard to read.

Software that comes as a prebuilt binary can also be a problem, because it expects libraries in places that don't exist on NixOS. I use nix-ld on my laptop for that, and it works most of the time.

In summary

For me it's been worth it. I can rebuild any of my machines from scratch, undo a bad change, and always know what's running where. If you're curious, reading other people's configs is a good way to start. There are a lot of them on GitHub.

In the next post I'll explain how I organise my config with den, which made a big difference once the server got bigger.

Thanks for reading!