Michael Stapelbergs Website

How my minimal, memory-safe Go rsync steers clear of vulnerabilities

2026-05-24T16:20:22+02:00

Back in January 2025, multiple different security researchers published a total of 6 security vulnerabilities in rsync, some of which allow arbitrary code execution and file leaks, so naturally I was wondering whether/how my gokrazy/rsync implementation was affected. Did implementing my own (compatible, but minimal) rsync in Go, a modern and memory-safe programming language, really rule out entire classes of security vulnerabilities?

This deep dive article was in the making since January 2025, but was delayed because we uncovered more unpublished vulnerabilities in the process! The “Security Vulnerabilities” section now covers all 12 vulnerabilities from the January 2025 batch and the May 2026 batch.

If you are running (upstream, samba) rsync in production, upgrade to version 3.4.3 or newer.

If you are running gokrazy/rsync in production, upgrade to version v0.3.3 or newer.

Feel free to skip over the nitty-gritty security issue details and jump directly to:

The verdict on whether using Go has helped.
The verdict on whether a minimal re-implementation like gokrazy/rsync helps.
My comparison with OpenBSD’s openrsync (written in C).
Defense in depth mechanisms one can use on Linux.
The conclusion.

Context: My own rsync

For context, I blogged about rsync, how I use it, and how it works back in June 2022. See also all posts tagged “rsync”.

The original motivation for writing my own rsync (back then only a server, today all directions are supported) was to provide the software packages of distri, my Linux distribution research project for fast package management, which I wanted to host on router7, my small home Linux+Go internet router, which in turn is built on gokrazy, my Go appliance platform.

I am still running multiple gokrazy/rsync servers for this original purpose, and also many others! Having rsync available as a primitive (that you can link into your Go programs!) is really nice.

Security Vulnerabilities

This article covers the following security vulnerabilities:

CVE-2024-12084 to 12088 (original report)
CVE-2024-12747 (discovered separately by Aleksei Gorban “loqpa”)
CVE-2026-29518 (discovered by Damien Neil and myself! and independently by Nullx3D)
CVE-2026-43617 to 43620
CVE-2026-45232

The first batch of the vulnerabilities above was announced on the oss-security mailing list, but note that the original report has more detail compared to the oss-security summaries!

The later vulnerabilities were announced via GitHub Security Advisories on the rsync project.

January 2025 batch

CVE-2024-12084: Heap Buffer Overflow (9.8)

Summary:

rsync performed insufficient validation: It read the (attacker-controlled) checksum length from the network and compared the length against MAX_DIGEST_LEN.
However, rsync’s data structures always declared a 16 byte buffer: char sum2[SUM_LENGTH]
- SUM_LENGTH is always 16 (bytes), which is sufficient to hold an MD4 or MD5 checksum.
- MAX_DIGEST_LEN used to be 16 (bytes), but can be larger when rsync is compiled with SHA256 or SHA512 checksum support.
Hence, the bounds check was ineffective! An attacker could write out of bounds.
This issue was introduced with commit ae16850 in September 2022, which added SHA256/SHA512 checksum support.

Click to expand the full description of the improper checksum length validation (quoting the Google Security report)

Stamp It! All Programs Must Report Their Version

2026-04-05T16:20:00+02:00

Recently, during a production incident response, I guessed the root cause of an outage correctly within less than an hour (cool!) and submitted a fix just to rule it out, only to then spend many hours fumbling in the dark because we lacked visibility into version numbers and rollouts… 😞

This experience made me think about software versioning again, or more specifically about build info (build versioning, version stamping, however you want to call it) and version reporting. I realized that for the i3 window manager, I had solved this problem well over a decade ago, so it was really unexpected that the problem was decidedly not solved at work.

In this article, I’ll explain how 3 simple steps (Stamp it! Plumb it! Report it!) are sufficient to save you hours of delays and stress during incident response.

Why are our versioning standards so low?!

Every household appliance has incredibly detailed versioning! Consider this dishwasher:

(Thank you Feuermurmel for sending me this lovely example!)

I observed a couple household appliance repairs and am under the impression that if a repair person cannot identify the appliance, they would most likely refuse to even touch it.

So why are our standards so low in computers, in comparison? Sure, consumer products are typically versioned somehow and that’s typically good enough (except for, say, USB 3.2 Gen 1×2!). But recently, I have encountered too many developer builds that were not adequately versioned!

Software Versioning

Unlike a physical household appliance with a stamped metal plate, software is constantly updated and runs in places and structures we often cannot even see.

Let’s dig into what we need to increase our versioning standard!

Usually, software has a name and some version number of varying granularity:

Chrome
Chrome 146
Chrome 146.0.7680.80
Chrome f08938029c887ea624da7a1717059788ed95034d-refs/branch-heads/7680_65@{#34}

All of these identify the Chrome browser on my computer, but each at different granularity.

All are correct and useful, depending on the context. Here’s an example for each:

“This works in Chrome for me, did you test in Firefox?”
“Chrome 146 contains broken middle-click-to-paste-and-navigate”
“I run Chrome 146.0.7680.80 and cannot reproduce your issue”
“Apply this patch on top of Chrome f08938029c887ea624da7a1717059788ed95034d-refs/branch-heads/7680_65@{#34} and follow these steps to reproduce: […]”

After creating the i3 window manager, I quickly learned that for user support, it is very valuable for programs to clearly identify themselves. Let me illustrate with the following case study.

Case Study: i3’s `--version` and `--moreversion`

When running i3 --version, you will see output like this:

% i3 --version
i3 version 4.24 (2024-11-06) © 2009 Michael Stapelberg and contributors

Each word was carefully deliberated and placed. Let me dissect:

i3 version 4.24: I could have shortened this to i3 4.24 or maybe i3 v4.24, but I figured it would be helpful to be explicit because i3 is such a short name. Users might mumble aloud “What’s an i-3-4-2-4?”, but when putting “version” in there, the implication is that i3 is some computer thing (→ a computer program) that exists in version 4.24.
(2024-11-06) is the release date so that you can immediately tell if “4.24” is recent.
© 2009 Michael Stapelberg signals when the project was started and who is the main person behind it.
and contributors gives credit to the many people who helped. i3 was never a one-person project; it was always a group effort.

When doing user support, there are a couple of questions that are conceptually easy to ask the affected user and produce very valuable answers for the developer:

Question: “Which version of i3 are you using?”
- Since i3 is not a typical program that runs in a window (but a window manager / desktop environment), there is no Help → About menu option.
- Instead, we started asking: What is the output of i3 --version?
Question: “Are you reporting a new issue or a preexisting issue? To confirm, can you try going back to the version of i3 you used previously?”. The technical terms for “going back” are downgrade, rollback or revert.
- Depending on the Linux distribution, this is either trivial or a nightmare.
- With NixOS, it’s trivial: you just boot into an older system “generation” by selecting that version in the bootloader. Or you revert in git, if your configs are version-controlled.
- With imperative Linux distributions like Debian Linux or Arch Linux, if you did not take a file system-level snapshot, there is no easy and reliable way to go back after upgrading your system. If you are lucky, you can just apt install the older version of i3. But you might run into dependency conflicts (“version hell”).
- I know that it is possible to run older versions of Debian using snapshot.debian.org, but it is just not very practical, at least when I last tried.
Can you check if the issue is still present in the latest i3 development version?
- Of course, I could also try reproducing the user issue with the latest release version, and then one additional time on the latest development version.
- But this way, the verification step moves to the affected user, which is good because it filters for highly-motivated bug reporters (higher chance the bug report actually results in a fix!) and it makes the user reproduce the bug twice, figuring out if it’s a flaky issue, hard-to-reproduce, if the reproduction instructions are correct, etc.
- A natural follow-up question: “Does this code change make the issue go away?” This is easy to test for the affected user who now has a development environment.

Based on my experiences with asking these questions many times, I noticed a few patterns in how these debugging sessions went. In response, I introduced another way for i3 to report its version in i3 v4.3 (released in September 2012): a --moreversion flag! Now I could ask users a small variation of the first question: What is the output of i3 --moreversion? Note how this also transfers well over spoken word, for example at a computer meetup:

Michael: Which version are you using?

User: How can I check?

Michael: Run this command: i3 --version

User: It says 4.24.

Michael: Good, that is recent enough to include the bug fix. Now, we need more version info! Run i3 --moreversion please and tell me what you see.

When you run i3 --moreversion, it does not just report the version of the i3 program you called, it also connects to the running i3 window manager process in your X11 session using its IPC (interprocess communication) interface and reports the running i3 process’s version, alongside other key details that are helpful to show the user, like which configuration file is loaded and when it was last changed:

% i3 --moreversion
Binary i3 version:  4.24 (2024-11-06) © 2009 Michael Stapelberg and…
Running i3 version: 4.24 (2024-11-06) (pid 2521)
Loaded i3 config:
  /home/michael/.config/i3/config (main)
  (last modified: 2026-03-15T23:09:27 CET, 1101585 seconds ago)

The i3 binary you just called:
/nix/store/0zn9r4263fjpqah6vdzlalfn0ahp8xc2-i3-4.24/bin/i3
The i3 binary you are running: i3

This might look like a lot of detail on first glance, but let me spell out why this output is such a valuable debugging tool:

Connecting to i3 via the IPC interface is an interesting test in and of itself. If a user sees i3 --moreversion output, that implies they will also be able to run debugging commands like (for example) i3-msg -t get_tree > /tmp/tree.json to capture the full layout state.
During a debugging session, running i3 --moreversion is an easy check to see if the version you just built is actually effective (see the Running i3 version line).
- Note that this is the same check that is relevant during production incidents: verifying that effectively running matches supposed to be running versions.
Showing the full path to the loaded config file will make it obvious if the user has been editing the wrong file. If the path alone is not sufficient, the modification time (displayed both absolute and relative) will flag editing the wrong file.

I use NixOS, BTW, so I automatically get a stable identifier (0zn9r4263fjpqah6vdzlalfn0ahp8xc2-i3-4.24) for the specific build of i3.

% ls -l $(which i3)
lrwxrwxrwx 1 root root 58 1970-01-01 01:00 /run/current-system/sw/bin/i3
-> /nix/store/0zn9r4263fjpqah6vdzlalfn0ahp8xc2-i3-4.24/bin/i3

To see the build recipe (“derivation” in Nix terminology) which produced this Nix store output (0zn9r4263…-i3-4.24), I can run nix derivation show:

% nix derivation show /nix/store/0zn9r4263fjpqah6vdzlalfn0ahp8xc2-i3-4.24
{
  "/nix/store/z7ly4kvgixf29rlz01ji4nywbajfifk4-i3-4.24.drv": {
[…]

Click here to expand the full nix derivation show output if you are curious

Coding Agent VMs on NixOS with microvm.nix

2026-02-01T09:00:00+01:00

I have come to appreciate coding agents to be valuable tools for working with computer program code in any capacity, such as learning about any program’s architecture, diagnosing bugs or developing proofs of concept. Depending on the use-case, reviewing each command the agent wants to run can get tedious and time-consuming very quickly. To safely run a coding agent without review, I wanted a Virtual Machine (VM) solution where the agent has no access to my personal files and where it’s no big deal if the agent gets compromised by malware: I can just throw away the VM and start over.

Instead of setting up a stateful VM and re-installing it when needed (ugh!), I prefer the model of ephemeral VMs where nothing persists on disk, except for what is explicitly shared with the host.

The microvm.nix project makes it easy to create such VMs on NixOS, and this article shows you how I like to set up my VMs.

Setting up microvm.nix

Let’s jump right in! The next sections walk you through how I set up my config.

Step 1: network prep

First, I created a new microbr bridge which uses 192.168.33.1/24 as IP address range and NATs out of the eno1 network interface. All microvm* interfaces will be added to that bridge:

systemd.network.netdevs."20-microbr".netdevConfig = {
  Kind = "bridge";
  Name = "microbr";
};

systemd.network.networks."20-microbr" = {
  matchConfig.Name = "microbr";
  addresses = [ { Address = "192.168.83.1/24"; } ];
  networkConfig = {
    ConfigureWithoutCarrier = true;
  };
};

systemd.network.networks."21-microvm-tap" = {
  matchConfig.Name = "microvm*";
  networkConfig.Bridge = "microbr";
};

networking.nat = {
  enable = true;
  internalInterfaces = [ "microbr" ];
  externalInterface = "eno1";
};

Step 2: `flake.nix`

Then, I added the microvm module as a new input to my flake.nix (check out the microvm.nix documentation for details) and enabled the microvm.nixosModules.host module on the NixOS configuration for my PC (midna). I also created a new microvm.nix file, in which I declare all my VMs. Here’s what my flake.nix looks like:

{
  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-25.11";
    };
    # For more recent claude-code
    nixpkgs-unstable = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    stapelbergnix = {
      url = "github:stapelberg/nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    zkjnastools = {
      url = "github:stapelberg/zkj-nas-tools";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    microvm = {
      url = "github:microvm-nix/microvm.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    home-manager = {
      url = "github:nix-community/home-manager/release-25.11";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    configfiles = {
      url = "github:stapelberg/configfiles";
      flake = false; # repo is not a flake
    };
  };

  outputs =
    {
      self,
      stapelbergnix,
      zkjnastools,
      nixpkgs,
      nixpkgs-unstable,
      microvm,
      home-manager,
      configfiles,
    }@inputs:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
      pkgs-unstable = import nixpkgs-unstable {
        inherit system;
        config.allowUnfree = true;
      };
    in
    {
      nixosConfigurations = {
        midna = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          specialArgs = { inherit inputs; };
          modules = [
            (import ./configuration.nix)
            stapelbergnix.lib.userSettings
            # Use systemd for network configuration
            stapelbergnix.lib.systemdNetwork
            # Use systemd-boot as bootloader
            stapelbergnix.lib.systemdBoot
            # Run prometheus node exporter in tailnet
            stapelbergnix.lib.prometheusNode
            zkjnastools.nixosModules.zkjbackup
            microvm.nixosModules.host
            ./microvm.nix
          ];
        };
      };
    };
}

Step 3: `microvm.nix`

The following microvm.nix declares two microvms, one for Emacs (about which I wanted to learn more) and one for Go Protobuf, a code base I am familiar with and can use to understand Claude’s capabilities:

{
  config,
  lib,
  pkgs,
  inputs,
  ...
}:

let
  inherit (inputs)
    nixpkgs-unstable
    stapelbergnix
    microvm
    configfiles
    home-manager
    ;

  microvmBase = import ./microvm-base.nix;
in
{
  microvm.vms.emacsvm = {
    autostart = false;
    config = {
      imports = [
        stapelbergnix.lib.userSettings
        microvm.nixosModules.microvm
        (microvmBase {
          hostName = "emacsvm";
          ipAddress = "192.168.83.6";
          tapId = "microvm4";
          mac = "02:00:00:00:00:05";
          workspace = "/home/michael/microvm/emacs";
          inherit
            nixpkgs-unstable
            configfiles
            home-manager
            stapelbergnix
            ;
        })
        ./microvms/emacs.nix
      ];
    };
  };

  microvm.vms.goprotobufvm = {
    autostart = false;
    config = {
      imports = [
        stapelbergnix.lib.userSettings
        microvm.nixosModules.microvm
        (microvmBase {
          hostName = "goprotobufvm";
          ipAddress = "192.168.83.7";
          tapId = "microvm5";
          mac = "02:00:00:00:00:06";
          workspace = "/home/michael/microvm/goprotobuf";
          inherit
            nixpkgs-unstable
            configfiles
            home-manager
            stapelbergnix
            ;
          extraZshInit = ''
            export GOPATH=$HOME/go
            export PATH=$GOPATH/bin:$PATH
          '';
        })
        ./microvms/goprotobuf.nix
      ];
    };
  };
}

Step 4: `microvm-base.nix`

The microvm-base.nix module takes these parameters and declares:

Network settings: I like using systemd-networkd(8) and systemd-resolved(8) .
Shared directories for:
- the workspace directory, e.g. ~/microvm/emacs
- the host’s Nix store, so the VM can access software from cache (often)
- this VM’s SSH host keys
- ~/claude-microvm, which is a separate state directory, used only on the microvms.
an 8 GB disk overlay (var.img), stored in /var/lib/microvms/
cloud-hypervisor (QEMU also works well!) as the hypervisor, with 8 vCPUs and 4 GB RAM.
A workaround for systemd trying to unmount /nix/store (which causes a deadlock).

Expand full microvm-base.nix code

Can I finally start using Wayland in 2026?

2026-01-04T08:55:00+01:00

Wayland is the successor to the X server (X11, Xorg) to implement the graphics stack on Linux. The Wayland project was actually started in 2008, a year before I created the i3 tiling window manager for X11 in 2009 — but for the last 18 years (!), Wayland was never usable on my computers. I don’t want to be stuck on deprecated software, so I try to start using Wayland each year, and this articles outlines what keeps me from migrating to Wayland in 2026.

Historical context

For the first few years, Wayland rarely even started on my machines. When I was lucky enough for something to show up, I could start some toy demo apps in the demo compositor Weston.

Around 2014, GNOME started supporting Wayland. KDE followed a few years later. Major applications (like Firefox, Chrome or Emacs) have been slower to adopt Wayland and needed users to opt into experimental implementations via custom flags or environment variables, until very recently, or — in some cases, like geeqie — still as of today.

Unfortunately, the driver support situation remained poor for many years. With nVidia graphics cards, which are the only cards that support my 8K monitor, Wayland would either not work at all or exhibit heavy graphics glitches and crashes.

In the 2020s, more and more distributions announced looking to switch to Wayland by default or even drop their X11 sessions, and RHEL is winding down their contributions to the X server.

Modern Linux distributions like Asahi Linux (for Macs, with their own GPU driver!) clearly consider Wayland their primary desktop stack, and only support X11 on a best-effort basis.

So the pressure to switch to Wayland is mounting! Is it ready now? What’s missing?

Making Wayland start

Hardware

I’m testing with my lab PC, which is a slightly upgraded version of my 2022 high-end Linux PC.

I describe my setup in more details in stapelberg uses this: my 2020 desk setup.

Most importantly for this article, I use a Dell 8K 32" monitor (resolution: 7680x4320!), which, in my experience, is only compatible with nVidia graphics cards (I try other cards sometimes).

Hence, both the lab PC and my main PC contain an nVidia GPU:

The lab PC contains a nVidia GeForce RTX 4070 Ti.
The main PC contains a nVidia GeForce RTX 3060 Ti.

(In case you’re wondering why I use the older card in my PC: I had a crash once where I suspected the GPU, so I switched back from the 4070 to my older 3060.)

nVidia driver support

For many years, nVidia drivers were entirely unsupported under Wayland.

Apparently, nVidia refused to support the API that Wayland was using, insisting that their EGLStreams approach was superior. Luckily, with nVidia driver 495 (late 2021), they added support for GBM (Generic Buffer Manager).

But, even with GBM support, while you could now start many Wayland sessions, the session wouldn’t run smoothly: You would see severe graphics glitches and artifacts, preventing you from getting any work done.

The solution for the glitches was explicit sync support: because the nVidia driver does not support implicit sync (like AMD or Intel), Wayland (and wlroots, and sway) needed to get explicit sync support.

Sway 1.11 (June 2025) and wlroots 0.19.0 are the first version with explicit sync support.

Not working: TILE support for 8K monitor

With the nVidia driver now working per se with Wayland, unfortunately that’s still not good enough to use Wayland in my setup: my Dell UP3218K monitor requires two DisplayPort 1.4 connections with MST (Multi Stream Transport) and TILE support. This combination worked just fine under X11 for the last 8+ years.

While GNOME successfully configures the monitor with its native resolution of 7680x4320@60, the monitor incorrectly shows up as two separate monitors in sway.

The reason behind this behavior is that wlroots does not support the TILE property (issue #1580 from 2019). Luckily, in 2023, contributor EBADBEEF sent draft merge request !4154, which adds support for the TILE property.

But, even with the TILE patch, my monitor would not work correctly: The right half of the monitor would just stay black. The full picture is visible when taking a screenshot with grim, so it seems like an output issue. I had a few exchanges about this with EBADBEEF starting in August 2025 (thanks for taking a look!), but we couldn’t figure out the issue.

A quarter later, I had made good experiences regarding debugging complex issues with the coding assistant Claude Code (Opus 4.5 at the time of writing), so I decided to give it another try. Over two days, I ran a number of tests to narrow down the issue, letting Claude analyze source code (of sway, wlroots, Xorg, mesa, …) and produce test programs that I could run manually.

Ultimately, I ended up with a minimal reproducer program (independent of Wayland) that shows how the SRC_X DRM property does not work on nVidia (but does work on Intel, for example!): I posted a bug report with a video in the nVidia forum and hope an nVidia engineer will take a look!

Crucially, with the bug now identified, I had Claude implement a workaround: copy the right half of the screen (at SRC_X=3840) to another buffer, and then display that buffer, but with SRC_X=0.

With that patch applied, for the first time, I can use Sway on my 8K monitor! 🥳

By the way, when I mentioned that GNOME successfully configures the native resolution, that doesn’t mean the monitor is usable with GNOME! While GNOME supports tiled displays, the updates of individual tiles are not synchronized, so you see heavy tearing in the middle of the screen, much worse than anything I have ever observed under X11. GNOME/mutter merge request !4822 should hopefully address this.

Software: NixOS

During 2025, I switched all my computers to NixOS. Its declarative approach is really nice for doing such tests, because you can reliably restore your system to an earlier version.

To make a Wayland/sway session available on my NixOS 25.11 installation, I added the following lines to my NixOS configuration file (configuration.nix):

# GDM display manager (can launch both X11/i3 and Wayland/Sway sessions)
services.displayManager.gdm.enable = true;
services.displayManager.gdm.autoSuspend = false;

# enable GNOME (for testing)
services.desktopManager.gnome.enable = true;

programs.sway = {
  enable = true;
  wrapperFeatures.gtk = true;
  extraOptions = [ "--unsupported-gpu" ];
};

I also added the following Wayland-specific programs to environment.systemPackages:

environment.systemPackages = with pkgs; [
  # …
  foot          # terminal emulator
  wtype         # replacement for xdotool type
  fuzzel        # fuzzy matching program starter
  wayland-utils # for wayland-info(1)
  gammastep     # redshift replacement
];

Note that activating this configuration kills your running X11 session, if any.

Just to be sure, I rebooted the entire machine after changing the configuration.

Experiment results

With this setup, I spent about one full work day in a Wayland session. Trying to actually get some work done uncovers issues that might not show in casual testing. Most of the day was spent trying to fix Wayland issues 😅. The following sections explain what I have learned/observed.

Desktop: i3 → sway

Many years ago, when Wayland became more popular, people asked on the i3 issue tracker if i3 would be ported to Wayland. I said no: How could I port a program to an environment that doesn’t even run on any of my computers? But also, I knew that with working a full-time job, I wouldn’t have time to be an early adopter and shape Wayland development.

This attitude resulted in Drew DeVault starting the Sway project around 2016, which aims to be a Wayland version of i3. I don’t see Sway as competition. Rather, I thought it was amazing that people liked the i3 project so much that they would go through the trouble of creating a similar program for other environments! What a nice compliment! 😊

Sway aims to be compatible with i3 configuration files, and it mostly is.

If you’re curious, here is what I changed from the Sway defaults, mostly moving key bindings around for the NEO keyboard layout I use, and configuring input/output blocks that I formerly configured in my ~/.xsession file:

my changes to the default Sway config

Self-hosting my photos with Immich

2025-11-29T08:22:05+01:00

For every cloud service I use, I want to have a local copy of my data for backup purposes and independence. Unfortunately, the gphotos-sync tool stopped working in March 2025 when Google restricted the OAuth scopes, so I needed an alternative for my existing Google Photos setup. In this post, I describe how I have set up Immich, a self-hostable photo manager.

Here is the end result: a few (live) photos from NixCon 2025:

Step 1. Hardware

I am running Immich on my Ryzen 7 Mini PC (ASRock DeskMini X600), which consumes less than 10 W of power in idle and has plenty of resources for VMs (64 GB RAM, 1 TB disk). You can read more about it in my blog post from July 2024:

Ryzen 7 Mini-PC makes a power-efficient VM host

When I saw the first reviews of the ASRock DeskMini X600 barebone, I was immediately interested in building a home-lab hypervisor (VM host) with it. Apparently, the DeskMini X600 uses less than 10W of power but supports latest-generation AMD CPUs like the Ryzen 7 8700G! Read more →

I installed Proxmox, an Open Source virtualization platform, to divide this mini server into VMs, but you could of course also install Immich directly on any server.

Step 2. Install Immich

I created a VM (named “photos”) with 500 GB of disk space, 4 CPU cores and 4 GB of RAM.

For the initial import, you could assign more CPU and RAM, but for normal usage, that’s enough.

I (declaratively) installed NixOS on that VM as described in this blog post:

How I like to install NixOS (declaratively)

For one of my network storage PC builds, I was looking for an alternative to Flatcar Container Linux and tried out NixOS again (after an almost 10 year break). There are many ways to install NixOS, and in this article I will outline how I like to install NixOS on physical hardware or virtual machines: over the network and fully declaratively. Read more →

Afterwards, I enabled Immich, with this exact configuration:

services.immich = {
  enable = true;
};

At this point, Immich is available on localhost, but not over the network, because NixOS enables a firewall by default. I could enable the services.immich.openFirewall option, but I actually want Immich to only be available via my Tailscale VPN, for which I don’t need to open firewall access — instead, I use tailscale serve to forward traffic to localhost:2283:

photos# tailscale serve --bg http://localhost:2283

Because I have Tailscale’s MagicDNS and TLS certificate provisioning enabled, that means I can now open https://photos.example.ts.net in my browser on my PC, laptop or phone.

Step 2. Initial photos import

At first, I tried importing my photos using the official Immich CLI:

% nix run nixpkgs#immich-cli -- login https://photos.example.ts.net secret
% nix run nixpkgs#immich-cli -- upload --recursive /home/michael/lib/photo/gphotos-takeout

Unfortunately, the upload was not running reliably and had to be restarted manually a few times after running into a timeout. Later I realized that this was because the Immich server runs background jobs like thumbnail creation, metadata extraction or face detection, and these background jobs slow down the upload to the extent that the upload can fail with a timeout.

The other issue was that even after the upload was done, I realized that Google Takeout archives for Google Photos contain metadata in separate JSON files next to the original image files:

Unfortunately, these files are not considered by immich-cli.

Luckily, there is a great third-party tool called immich-go, which solves both of these issues! It pauses background tasks before uploading and restarts them afterwards, which works much better, and it does its best to understand Google Takeout archives.

I ran immich-go as follows and it worked beautifully:

% immich-go \
  upload \
  from-google-photos \
  --server=https://photos.example.ts.net \
  --api-key=secret \
  ~/Downloads/takeout-*.zip

Step 3. Install the Immich iPhone App

My main source of new photos is my phone, so I installed the Immich app on my iPhone, logged into my Immich server via its Tailscale URL and enabled automatic backup of new photos via the icon at the top right.

I am not 100% sure whether these settings are correct, but it seems like camera photos generally go into Live Photos, and Recent should cover other files…?!

If anyone knows, please send an explanation (or a link!) and I will update the article.

I also strongly recommend to disable notifications for Immich, because otherwise you get notifications whenever it uploads images in the background. These notifications are not required for background upload to work, as an Immich developer confirmed on Reddit. Open Settings → Apps → Immich → Notifications and un-tick the permission checkbox:

Step 4. Backup

Immich’s documentation on backups contains some good recommendations. The Immich developers recommend backing up the entire contents of UPLOAD_LOCATION, which is /var/lib/immich on NixOS. The backups subdirectory contains SQL dumps, whereas the 3 directories upload, library and profile contain all user-uploaded data.

Hence, I have set up a systemd timer that runs rsync to copy /var/lib/immich onto my PC, which is enrolled in a 3-2-1 backup scheme.

What’s missing?

Immich (currently?) does not contain photo editing features, so to rotate or crop an image, I download the image and use GIMP.

To share images, I still upload them to Google Photos (depending on who I share them with).

Why Immich instead of…?

The two most promising options in the space of self-hosted image management tools seem to be Immich and Ente.

I got the impression that Immich is more popular in my bubble, and Ente made the impression on me that its scope is far larger than what I am looking for:

Ente is a service that provides a fully open source, end-to-end encrypted platform for you to store your data in the cloud without needing to trust the service provider. On top of this platform, we have built two apps so far: Ente Photos (an alternative to Apple and Google Photos) and Ente Auth (a 2FA alternative to the deprecated Authy).

I don’t need an end-to-end encrypted platform. I already have encryption on the transit layer (Tailscale) and disk layer (LUKS), no need for more complexity.

Conclusion

Immich is a delightful app! It’s very fast and generally seems to work well.

The initial import is smooth, but only if you use the right tool. Ideally, the official immich-cli could be improved. Or maybe immich-go could be made the official one.

I think the auto backup is too hard to configure on an iPhone, so that could also be improved.

But aside from these initial stumbling blocks, I have no complaints.

My impressions of the MacBook Pro M4

2025-10-31T11:04:59+01:00

I have been using a MacBook Pro M4 as my portable computer for the last half a year and wanted to share a few short impressions. As always, I am not a professional laptop reviewer, so in this article you won’t find benchmarks, just subjective thoughts!

Back in 2021, I wrote about the MacBook Air M1, which was the first computer I used that contained Apple’s own ARM-based CPU. Having a silent laptop with long battery life was a game-changer, so I wanted to keep those properties.

When the US government announced tariffs, I figured I would replace my 4-year old MacBook Air M1 with a more recent model that should last a few more years. Ultimately, Apple’s prices remained stable, so, in retrospect, I could have stayed with the M1 for a few more years. Oh well.

The nano-textured display

I went to the Apple Store to compare the different options in person. Specifically, I was curious about the display and whether the increased weight and form factor of the MacBook Pro (compared to a MacBook Air) would be acceptable. Another downside of the Pro model is that it comes with a fan, and I really like absolutely quiet computers. Online, I read from other MacBook Pro owners that the fan mostly stays off.

In general, I would have preferred to go with a MacBook Air because it has enough compute power for my needs and I like the case better (no ventilation slots), but unfortunately only the MacBook Pro line has the better displays.

Why aren’t all displays nano-textured? The employee at the Apple Store presented the trade-off as follows: The nano texture display is great at reducing reflections, at the expense of also making the picture slightly less vibrant.

I could immediately see the difference when placing two laptops side by side: The bright Apple Store lights showed up very prominently on the normal display (left), and were almost not visible at all on the nano texture display (right):

Personally, I did not perceive a big difference in “vibrancy”, so my choice was clear: I’ll pick the MacBook Pro over the MacBook Air (despite the weight) for the nano texture display!

After using the laptop in a number of situations, I am very happy with this choice. In normal scenarios, I notice no reflections at all (where my previous laptop did show reflections!). This includes using the laptop on a train (next to the window), or using the laptop outside in daylight.

Specs: M4 or M4 Pro?

(When I chose the new laptop, Apple’s M4 chips were current. By now, they have released the first devices with M5 chips.)

I decided to go with the MacBook Pro with M4 chip instead of the M4 Pro chip because I don’t need the extra compute, and the M4 needs less cooling — the M4 Pro apparently runs hotter. This increases the chance of the fan staying off.

Here are the specs I ended up with:

14" Liquid Retina XDR Display with nano texture
Apple M4 Chip (10 core CPU, 10 core GPU)
32 GB RAM (this is the maximum!), 2 TB SSD (enough for this computer)

Impressions

One thing I noticed is that the MacBook Pro M4 sometimes gets warm, even when it is connected to power, but is suspended to RAM (and has been fully charged for hours). I’m not sure why.

Luckily, the fan indeed stays silent. I think I might have heard it spin up once in half a year or so?

The battery life is amazing! The previous MacBook Air M1 had amazing all-day battery life already, and this MacBook Pro M4 lasts even longer. For example, watching videos on a train ride (with VLC) for 3 hours consumed only 10% of battery life. I generally never even carry the charger.

Because of that, Apple’s re-introduction of MagSafe, a magnetic power connector (so you don’t damage the laptop when you trip over it), is nice-to-have but doesn’t really make much of a difference anymore. In fact, it might be better to pack a USB-C cable when traveling, as that makes you more flexible in how you use the charger.

120 Hz display

I was curious whether the 120 Hz display would make a difference in practice. I mostly notice the increased refresh rate when there are animations, but not, for example, when scrolling.

One surprising discovery (but obvious in retrospect) is that even non-animations can become faster. For example, when running a Go web server on localhost, I noticed that navigating between pages by clicking links felt faster on the 120 Hz display!

The following illustration shows why that is, using a page load that takes 6ms of processing time. There are three cases (the illustration shows an average case and the worst case):

Best case: Page load finishes just before the next frame is displayed: no delay.
Worst case: Page load finishes just after a frame is displayed: one frame of delay.
Most page loads are somewhere in between. We’ll have 0.x to 1.0 frames of delay

As you can see, the waiting time becomes shorter when going from 60 Hz (one frame every 16.6ms) to 120 Hz (one frame every 8.3ms). So if you’re working with a system that has <8ms response times, you might observe actions completing (up to) twice as fast!

I don’t notice going back to 60 Hz displays on computers. However, on phones, where a lot more animations are a key part of the user experience, I think 120 Hz displays are more interesting.

Conclusion

My ideal MacBook would probably be a MacBook Air, but with the nano-texture display! :)

I still don’t like macOS and would prefer to run Linux on this laptop. But Asahi Linux still needs some work before it’s usable for me (I need external display output, and M4 support). This doesn’t bother me too much, though, as I don’t use this computer for serious work.

NixCon 2025 Trip Report 🐝

2025-09-21T09:34:00+02:00

I liked the NixOS meetup earlier this year, and at the end of the meetup they told everyone about NixCon 2025, which would be happening in Switzerland this year, at the very same location, the University Of Applied Sciences OST in Rapperswil, so I decided to go! In this trip report, I want to give you a rough impression of how I experienced this awesome conference :)

The bee in the title is a NixCon inside joke ;)

Friday

I arrived at about 09:30 on a rainy Friday morning, meaning I hurried from the train station into OST building 1 to show my ticket QR code and pick up my conference badge and custom name tag that I pre-ordered. The custom ones have your name engraved and come with a strong magnet to attach them to your clothes:

After grabbing a bite to eat, I headed to the main lecture hall for the opening session. Prof. Dr. Farhad Mehta from OST, as well as the entire NixCon orga team, welcomed the 450 registered attendees to the 10th NixCon! I recognized many familiar faces from the Nix meetup, but many hands went up when the audience was asked for whom it was the first time at NixCon, or in Switzerland in general.

I want to thank Prof. Mehta in particular for making possible such meetups and events! 👏

If you work at a university, school or other organisation that has access to rooms, consider offering to host a meetup (on a regular basis, or even just once)! Locations are always hard to find, so offering a space is a great contribution to Open Source.

“What if GitHub Actions were local-first and built using Nix?”

The first technical talk of the day was “What if GitHub Actions were local-first and built using Nix?” by Domen Kožar, the person behind cachix.org, which is a hosted Nix cache. The talk pitched cloud.devenv.sh, which is a Nix-based CI solution (like GitHub Actions) using devenv.

By using this solution, you solve the problem that you can’t easily / completely run GitHub Actions locally (yes, we all know about act), and you get to (?) write Nix configs instead of YAML configs.

The solution seems nice, but I found the talk a little unstructured because the presenter jumped around between slides so much. One crucial question was left unanswered: How do you integrate this custom solution with your GitHub projects? To me, diverging from the default way of configuring GitHub Actions does not seem worth it for my projects. YMMV.

→ watch the recording (46 minutes) on media.ccc.de

“Rewriting the Hydra Queue Runner in Rust”

Next up: “Rewriting the Hydra Queue Runner in Rust” by Simon Hauser from Helsinki Systems, a small German software company. Hydra is the component in the NixOS infrastructure which schedules builds: when nixpkgs changes, this is the component that runs the build whose result ends up on cache.nixos.org (the Debian equivalent is buildd).

Simon explained that bottlenecks in the current queue runner result in stranding of infrastructure: the project has machines available that it cannot use fully. He outlined how they replaced a crufty SSH-based automation with a well-designed gRPC protocol. I got the impression that a group of people was involved in developing and reviewing this design, which is a great sign for a healthy project.

One thing that was unfortunately missing from the talk were metrics. It would have been great to see a few graphs that illustrate just how much better the rewritten queue runner is.

Currently, the new queue runner is already used for Nix Community builds, but not yet in production for NixOS itself. Hopefully soon, though!

→ watch the recording (27 minutes) on media.ccc.de

“You can’t spell “devshell” without “hell””

This talk was presented by Zach Mitchell from Flox, which is a Nix-based dev environment solution. Thus far, I use nix-shell or nix develop (see Development shells with Nix: four quick examples), so I was curious what I’d learn from this talk.

Zach explained that both, nix-shell and nix develop were originally written to debug Nix package builds, not to provide general-purpose development environments. For users, this manifests as not being able to use your favorite shell — nix develop only supports Bash. One might read about nix develop -c exec , but that’s wrong, because then the shell’s RC files run after Nix setup, possibly destroying parts of the setup.

One interesting thing I learnt is that the Nix garbage collector scans /proc to avoid removing Nix store paths that are still needed by running processes.

Zach mentioned https://github.com/zmitchell/proctrace, which is a bpftrace-based profiler that tracks forks/execs and generates gantt chart syntax of the timing. Sounds cool, but is unfortunately broken right now…? Too bad.

→ watch the recording (45 minutes) on media.ccc.de

“The Nix Binary Cache and AWS”

In this fireside chat, Tarus Balog shared how he ended up at AWS after 20 years of Open Source, and how his team wants to give back to the community. One specific way in which they’re doing that is by hosting cache.nixos.org.

→ watch the recording (24 minutes) on media.ccc.de

“Nix-based development environments at Shopify (reprise)”

Josh Heinrichs from Shopify shared how they adopted Nix (again!), and I think real-world enterprise adoption stories like these are very interesting.

In summary, Shopify had a dev command (since 2016), which offered declarative configuration and then dispatched to apt (Linux) or homebrew (macOS). In the first attempt to move to Nix, the effort didn’t reach stable footing (some folks couldn’t use it yet) and then a company-wide shift to cloud development happened, where the easier solution was to “just use ubuntu”.

A few years in, folks are apparently not so happy with the cloud development environments and one day, Shopify CEO Tobias Lütke finds devenv, which is a Nix-based solution that is remarkably similar to Shopify’s dev. So Tobi adopts devenv for one of their services and becomes supportive of using Nix. This time around, they spend a lot more time on a successful rollout within the organization, meaning incremental adoption, getting all stakeholders on board, etc.

The takeaway is that one specific, well-supported use-case can be the adoption driver. And once you have your development environments on a Nix-based solution, you can more easily adopt other parts of the ecosystem as well.

→ watch the recording (19 minutes) on media.ccc.de

“My first Nix Aha!: A Newcomer’s Perspective”

In a similar spirit to the Shopify talk, Kavisha Kumar from ASML shared how she got into Nix after seeing a colleague use nix-shell to obtain a clean development shell.

Kavisha spent a lot of time at ASML to teach others about why and how to use Nix. She shared a number of nice metaphors that explained Nix concepts through the subject area of video gaming.

I think many people are excited about Nix, but have trouble conveying that excitement to others. Kavisha showed us a good way that worked for her.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day was filled with lightning talks.

Cole Mickens from Determinate Systems explained what features they are currently shipping in their downstream distribution “Determinate Nix” (features will be upstreamed): lazy trees (a performance optimization for evaluating Flakes), parallel evaluation (brings evaluation times down from 16s to 7s) and a native Linux builder for mac. Next up are Flake Schemas, which I haven’t read about yet.

Yvan Sraka from Numtide, a Nix and DevOps consultancy, showed how he manages Linux machines for friends and family with NixOS. He has his own configuration layer on top of NixOS and only uses the system as a base. Most actual programs are used through AppImage, Flatpaks, envfs and nix-ld. The latter two are solutions to use FHS based programs (those that expect /usr/bin and other standard locations to be present) on non-FHS systems like NixOS. I had heard of nix-ld before, but not of envfs.

Jacek Galowicz from Nixcademy showed how to use systemd-sysupdate and systemd-repart to implement A/B style updates with NixOS and systemd. It’s great to see that this technique is more and more mainstream, as I am also using A/B style updating successfully in gokrazy.

Saturday

The weather on Saturday was a lot better, so I made sure to get a seat with a view of Lake Zürich:

“The bikes have been shed: The official Nix formatter”

In this talk, Silvan Mosberger from Tweag (and one of the main NixCon organizers!), explains how the official formatting tool for .nix files came to be.

I was delighted to hear gofmt, the official Go formatter, being mentioned as a source of inspiration. Just like in other language ecosystems, introducing uniform formatting eliminates time-consuming back-and-forth in code review over adhering to coding style. Unfortunately, the formatting folks did not replicate one key aspect to gofmt’s success: gofmt has no options. As the famous Go proverb goes:

Gofmt’s style is no one’s favorite, yet gofmt is everyone’s favorite!

Meaning that it’s more important that everyone uses the same style, compared to everyone being able to express their personal style preferences.

→ watch the recording (20 minutes) on media.ccc.de

“Mastering NixOS Integration Tests: Advanced Techniques for Fast and Robust Multi-VM Tests”

In this two-hour workshop, Jacek Galowicz from Nixcademy, who is not only a Nix teacher, but also happens to be the maintainer of the NixOS integration test driver, shows us how to write complex integration tests with a few lines of Nix and Python.

Jacek showed an integration test example: a Bittorrent service, consisting of tracker, clients, firewalls and multiple networks! Nixpkgs contains over 1000 such integration tests, and running one on your laptop is easy.

The various ways to debug your tests seem pretty cool: using vsock instead of port forwardings, and enabling a debug hook that will make a failed test hang and wait to be debugged.

I thought this was a great overview and Jacek is an engaging teacher. I would recommend booking his classes!

“When Not to Nix: Working with External Config and SOPS Nix”

Ryota spoke about when to use Nix and when not to use Nix. For example, you could manage your dotfiles (config files) with Nix, or you could decide not to. Having recently migrated more and more machines and configurations to Nix, I found myself agreeing with this talk: It’s important to understand what you’ll get out of declaratively or statefully managed configs, and when which approach is better.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day I spent in lightning talks, some of which were sponsored talk slots. I learnt about, in no particular order:

Cloud Hypervisor, a KVM based hypervisor like qemu, but written in Rust.
nixbuild.net, a pay-as-you-go offering for extra build capacity you can rent. On Sunday I heard someone say that their company is using nixbuild.net and it’s very smooth.
NixCI, a Nix-based hosted CI. So, the cloud.devenv.sh service we heard about on Friday is a competitor to this service.
Nix in the Wild is an effort by Flox where they do 45-60 minute interviews about Nix success stories. This might help you convince folks in your organization.
clan is a fleet management solution.
NovaCustom, a one-person laptop/PC company. The laptops come with coreboot and work with NixOS.
ExpressVPN is migrating their internal server setup (TrustedServer) from Debian to NixOS! Deploying weekly in 105+ countries.
Cyberus, a German company, is offering NixOS LTS releases, compliant with the EU Cyber Resilience Act obligations.
David’s styx project is a more bandwidth-efficient download mechanism for NixOS updates. This uses EROFS, which seems like an interesting alternative to SquashFS images.

After all the talks, we met outside for a group picture followed by barbecue at the lake:

NixCon 2025 by Arik Grahl. Licensed under CC BY-SA 4.0.

Sunday

Before the conference, I wasn’t sure if I would even bother showing up for Sunday (Hack day), but on Sunday, I was like “of course!”, and it was a great decision!

Many people were still around and were working on their projects. It felt like the answer to any Nix question was just one chat message away — there was expertise and helping hands from many parts of the project.

I ended up meeting a couple of people I only knew from online interactions before, and we also talked a lot about meetups. Now, I am invited to multiple meetups to give a talk :D

Conclusion

This was a wonderful conference! The orga team and all contributors did a great job!

As always, the OST in Rapperswil is a great venue for Open Source events.

Ticket sales and talk submission / scheduling were done using the Pretix and Pretalx Open Source systems, which makes me proud to have contributed to Pretix.

The selection of talks was great: Some deeply technical, some covering only the human side of things, and many somewhere in between. I got the impression that all the presenters I saw genuinely cared about their topic, so the overall energy was very good!

(You can watch the talk recordings at media.ccc.de: NixCon 2025.)

Also outside of the talks, I had many friendly interactions and interesting conversations. There is a lot of interest and adoption of Nix, which is great to see!

The production level of the conference was very high for such a volunteer-driven event. For example, the very cool sounding break music between talks was created specifically for NixCon: “Lava” by tonstr.studio. Similarly, the welcome bag contained dark Swiss chocolate, specifically made for NixCon (see picture below). I don’t even like dark chocolate, but this one was delicious!

Thanks again to all helpers, and I look forward to coming back soon!

Bye Intel, hi AMD! I’m done after 2 dead Intels

2025-09-07T08:33:00+02:00

The Intel 285K CPU in my high-end 2025 Linux PC died again! 😡 Notably, this was the replacement CPU for the original 285K that died in March, and after reading through the reviews of Intel CPUs on my electronics store of choice, many of which (!) mention CPU replacements, I am getting the impression that Intel’s current CPUs just are not stable 😞. Therefore, I am giving up on Intel for the coming years and have bought an AMD Ryzen 9950X3D CPU instead.

What happened? Or: the batch job of death

On the 9th of July, I set out to experiment with layout-parser and tesseract in order to convert a collection of scanned paper documents from images into text.

I expected that offloading this task to the GPU would result in a drastic speed-up, so I attempted to build layout-parser with CUDA. Usually, it’s not required to compile software yourself on NixOS, but CUDA is non-free, so the default NixOS cache does not compile software with CUDA. (Tip: Enable the Nix Community Cache, which contains prebuilt CUDA packages, too!)

This lengthy compilation attempt failed with a weird symptom: I left for work, and after a while, my PC was no longer reachable over the network, but fans kept spinning at 100%! 😳 At first, I suspected a Linux bug, but now I am thinking this was the first sign of the CPU being unreliable.

When the CUDA build failed, I ran the batch job without GPU offloading instead. It took about 4 hours and consumed roughly 300W constantly. You can see it on this CPU usage graph (screenshot of a Grafana dashboard showing metrics collected by Prometheus):

On the evening of the 9th, the computer still seemed to work fine.

But the next day, when I wanted to wake up my PC from suspend-to-RAM as usual, it wouldn’t wake up. Worse, even after removing the power cord and waiting a few seconds, there was no reaction to pressing the power button.

Later, I diagnosed the problem to either the mainboard and/or the CPU. The Power Supply, RAM and disk all work with different hardware. I ended up returning both the CPU and the mainboard, as I couldn’t further diagnose which of the two is broken.

To be clear: I am not saying the batch job killed the CPU. The computer was acting strangely in the morning already. But the batch job might have been what really sealed the deal.

No, it wasn’t the heat wave

Tom’s Hardware recently reported that “Intel Raptor Lake crashes are increasing with rising temperatures in record European heat wave”, which prompted some folks to blame Europe’s general lack of Air Conditioning.

But in this case, I actually did air-condition the room about half-way through the job (at about 16:00), when I noticed the room was getting hot. Here’s the temperature graph:

I would say that 25 to 28 degrees celsius are normal temperatures for computers.

I also double-checked if the CPU temperature of about 100 degrees celsius is too high, but no: this Tom’s Hardware article shows even higher temperatures, and Intel specifies a maximum of 110 degrees. So, running at “only” 100 degrees for a few hours should be fine.

Lastly, even if Intel CPUs were prone to crashing under high heat, they should never die.

Which AMD CPU to buy?

I wanted the fastest AMD CPU (for desktops, not for servers), which currently is the Ryzen 9 9950X, but there is also the Ryzen 9 9950X3D, a variant with 3D V-Cache. Depending on the use-case, the variant with or without 3D V-Cache is faster, see the comparison on Phoronix.

Ultimately, I decided for the 9950X3D model, not just because it performs better in many of the benchmarks, but also because Linux 6.13 and newer let you control whether to prefer the CPU cores with larger V-Cache or higher frequency, which sounds like an interesting capability: By changing this setting, maybe one can see how sensitive certain workloads are to extra cache.

Aside from the CPU, I also needed a new mainboard (for AMD’s socket AM5), but I kept all the other components. I ended up selecting the ASUS TUF X870+ mainboard. I usually look for low power usage in a mainboard, so I made sure to go with an X870 mainboard instead of an X870E one, because the X870E has two chipsets (both of which consume power and need cooling)! Given the context of this hardware replacement, I also like the TUF line’s focus on endurance…

Performance

The performance of the AMD 9950X3D seems to be slightly better than the Intel 285K:

Workload	12900K (2022)	285K (2025)	9950X3D (2025)
build Go 1.24.3	≈35s	≈26s	≈24s
gokrazy/rsync tests	≈0.5s	≈0.4s	≈0.5s
gokrazy Linux compile	3m 13s	2m 7s	1m 56s

In case you’re curious, the commands used for each workload are:

cd src; ./make.bash
make test
gokr-rebuild-kernel -cross=arm64

(I have not included the gokrazy UEFI integration tests because I think there is an unrelated difference that prevents comparison of my old results with how the test runs currently.)

Power consumption

In my high-end 2025 Linux PC I explained that I chose the Intel 285K CPU for its lower idle power consumption, and some folks were skeptical if AMD CPUs are really worse in that regard.

Having switched between 3 different PCs, but with identical peripherals, I can now answer the question of how the top CPUs differ in power consumption!

I picked a few representative point-in-time power values from a couple of days of usage:

CPU	Mainboard	idle power	idle power with monitor
Intel 12900k	ASUS PRIME Z690-A	40W	60W
Intel 285k	ASUS PRIME Z890-P	46W	65W
AMD 9950X3D	ASUS TUF GAMING X870-PLUS WIFI	55W	80W

Looking at two typical evenings, here is the power consumption of the Intel 285K (measured using a myStrom WiFi switch smart plug, which comes with a REST API):

…and here is the same PC setup, but with the AMD 9950X3D:

I get the general impression that the AMD CPU has higher power consumption in all regards: the baseline is higher, the spikes are higher (peak consumption) and it spikes more often / for longer.

Looking at my energy meter statistics, I usually ended up at about 9.x kWh per day for a two-person household, cooking with induction.

After switching my PC from Intel to AMD, I end up at 10-11 kWh per day.

Conclusion

I started buying Intel CPUs because they allowed me to build high-performance computers that ran Linux flawlessly and produced little noise. This formula worked for me over many years:

Back in 2008, I bought a mobile Intel CPU in a desktop case (article in German).
Then, in 2012, I could just buy a regular Intel CPU (i7-2600K) for my Linux PC, because they had gotten so much better in terms of power saving.
Over the years, I bought an i7-8700K, and later an i9-9900K.
The last time this formula worked out for me was with my 2022 high-end Linux PC.

On the one hand, I’m a little sad that this era has ended. On the other hand, I have had a soft spot for AMD since I had one of their K6 CPUs in one of my early PCs and in fact, I have never stopped buying AMD CPUs (e.g. for my Ryzen 7-based Mini Server).

Maybe AMD could further improve their idle power usage in upcoming models? And, if Intel survives for long enough, maybe they succeed at stabilizing their CPU designs again? I certainly would love to see some competition in the CPU market.

Secret Management on NixOS with sops-nix

2025-08-24T09:56:00+02:00

Passwords and secrets like cryptographic key files are everywhere in computing. When configuring a Linux system, sooner or later you will need to put a password somewhere — for example, when I migrated my existing Linux Network Storage (NAS) setup to NixOS, I needed to specify the desired Samba passwords in my NixOS config (or manage them manually, outside of NixOS). For personal computers, this is fine, but if the goal is to share system configurations (for example in a Git repository), we need a different solution: Secret Management.

What is Secret Management?

The basic idea behind Secret Management systems is to encrypt the secrets at rest, meaning if somebody clones the git repository containing your NixOS system configurations, they cannot access (and therefore, also not deploy) the encrypted secrets.

Conceptually, we need to:

Encrypt the secrets such that the target system can decrypt them.
Encrypt the secrets such that other people working on this config can decrypt them.
Have the target system decrypt secrets at runtime.
Tell our software where to access the decrypted secrets.

sops-nix setup

In this article, I will show how to accomplish the above using sops-nix. Here’s a quick overview of the three different building blocks we will use:

sops is a tool to version-control secrets in git, in their encrypted form.
- sops makes it easy to re-encrypt these secrets when adding/removing authorized keys.
- sops is very flexible and can work with tons of other tools/providers.
sops-nix provides a way to integrate sops with Nix/NixOS
Using sops with age(1) allows us to use our existing SSH private key (humans) or SSH host private key (machines) instead of managing a separate set of key files.

You might wonder why I chose sops-nix over agenix, the other contender? The instructions for setting up sops-nix made more sense to me when I first looked at it, and I wanted to have the option to use sops in other ways, not just with age. If you’re curious about agenix, check out Andreas Gohr’s blog post about agenix.

Step 1. Preparation

I ran the following instructions on an Arch Linux machine on which I installed the Nix tool and enabled Nix Flakes. Follow the link for instructions also for other systems like Debian or Fedora.

Step 2. Obtain an age identity from your personal SSH key

I don’t want to manage an extra key file, so I’ll use ssh-to-age to derive a key from my SSH private key file, which I already take good care of to back up:

midna % mkdir -p $HOME/.config/sops/age/
midna % read -s SSH_TO_AGE_PASSPHRASE; export SSH_TO_AGE_PASSPHRASE
midna % nix run nixpkgs#ssh-to-age -- \
  -private-key \
  -i $HOME/.ssh/id_ed25519 \
  -o $HOME/.config/sops/age/keys.txt

(The SSH_TO_AGE_PASSPHRASE option is documented in the ssh-to-age README.)

To display the age recipient (public key) of this age identity (private key), I used:

midna % nix shell nixpkgs#age
midna 2 % age-keygen -y $HOME/.config/sops/age/keys.txt
age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf

Step 3. Obtain an age recipient for the remote machine

Similarly, I will derive an age recipient from the SSH host key of the remote system:

batchn % cat /etc/ssh/ssh_host_ed25519_key.pub | nix run nixpkgs#ssh-to-age
age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k

Step 4. Configure sops for your git repository

In my git repository (nix-configs), I have one subdirectory per NixOS system, i.e. tree(1) shows:

├── batchn
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
│   ├── secrets
│   │   └── example.yaml
├── wiki
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
…

In the root of the git repository (next to the batchn directory), I create .sops.yaml like so:

keys:
  - &admin_michael age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf
  - &server_batchn age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k
# …more server keys go here…
creation_rules:
  - path_regex: batchn/secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
    - age:
      - *admin_michael
      - *server_batchn

The more systems I manage, the more keys and creation_rules I will need to configure.

The creation rules tell sops which keys to use when encrypting a file. In my setups, I typically use only a single file per system, but I could imagine splitting out some secrets into a separate file if I wanted to collaborate with someone on just one aspect of the system.

Step 5. Manage some secrets with sops

Now that we told sops which recipients to encrypt for, we can decrypt and edit secrets/example.yaml in our configured editor by running:

midna ~/nix-configs/batchn % nix run nixpkgs#sops secrets/example.yaml

The simplest key file contains just one key, for example:

api-key: hello world :)

After saving and exiting your editor, sops will update the encrypted secrets/example.yaml.

Step 6. Configure sops in NixOS

Now, we need to reference the encrypted file in NixOS and enable sops-nix integration to make the decrypted secrets available on the system.

In flake.nix, I added sops-nix to the inputs section and added the NixOS module. I show the entire diff because the places where the lines go are just as important as what the lines say:

--- c/batchn/flake.nix
+++ i/batchn/flake.nix
@@ -1,85 +1,93 @@
 {
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

     disko.url = "github:nix-community/disko";
     # Use the same version as nixpkgs
     disko.inputs.nixpkgs.follows = "nixpkgs";

     stapelbergnix.url = "github:stapelberg/nix";

     zkjnastools.url = "github:stapelberg/zkj-nas-tools";

+    sops-nix = {
+      url = "github:Mic92/sops-nix";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
   };

   outputs =
     {
       nixpkgs,
       disko,
       stapelbergnix,
       zkjnastools,
+      sops-nix,
       ...
     }:
     let
       system = "x86_64-linux";
       pkgs = import nixpkgs {
         inherit system;
         config.allowUnfree = false;
       };
     in
     {
       nixosConfigurations.batchn = nixpkgs.lib.nixosSystem {
         inherit system;
         inherit pkgs;
         modules = [
           disko.nixosModules.disko
           ./configuration.nix
           stapelbergnix.lib.userSettings
           # Use systemd for network configuration
           stapelbergnix.lib.systemdNetwork
           # Use systemd-boot as bootloader
           stapelbergnix.lib.systemdBoot
           # Run prometheus node exporter in tailnet
           stapelbergnix.lib.prometheusNode
           zkjnastools.nixosModules.zkjbackup
+          sops-nix.nixosModules.sops
         ];
       };
       formatter.${system} = pkgs.nixfmt-tree;
     };
 }

Then, in configuration.nix, we tell sops-nix to use the SSH host key as identity, where sops will find our secrets and which secrets sops-nix should realize on the remote system:

  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  sops.defaultSopsFile = ./secrets/example.yaml;
  sops.secrets."api-key" = { };

After deploying, we can access the secret on the running system:

batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Of course, even after rebooting the machine, the secrets remain available without a re-deploy:

batchn ~ % uptime
 22:09:23  up   0:00,  1 user,  load average: 0,32, 0,08, 0,03
batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Usage Examples

Now that we have secrets stored in files under /run/secrets, how can we use these secrets?

The following sections show a few common ways.

Usage Example: command-line flags (ExecStart wrapper)

Let’s assume you have deployed a custom Go server as a systemd service on NixOS as follows, and you want to start managing the cleartext secret passed via the -securecookie_hash_key and -securecookie_block_key command-line flags:

{
  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="some-secret-key" \
          -securecookie_block_key="a-different-secret-key"
      '';
    };
  };
}

With the following sops secrets:

fortuneserver:
    securecookie_hash_key: some-secret-key
    securecookie_block_key: a-different-secret-key

…we need to adjust our NixOS config to read these secret files at runtime. Because the ExecStart directive is interpreted by systemd and not passed through a shell, we use the writeShellScript helper and then just cat the files:

{
  sops.secrets."fortuneserver/securecookie_hash_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };
  sops.secrets."fortuneserver/securecookie_block_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };

  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = pkgs.writeShellScript "fortuneserver-execstart" ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="$(cat /run/secrets/fortuneserver/securecookie_hash_key)" \
          -securecookie_block_key="$(cat /run/secrets/fortuneserver/securecookie_block_key)"
      '';
    };
  };
}

Usage Example: environment variable files

What if the service in question does not use command-line flags, but environment variables for configuring secrets? We can put an environment variable file into a sops-managed secret:

translate-fe:
    env: |
        DEEPL_AUTH_KEY=my-deepl-key

…and then we make systemd apply these environment variables from the secrets file:

{
  sops.secrets."translate-fe/env" = {
    owner = "translatefe";
    restartUnits = [ "translate-fe.service" ];
  };

  systemd.services.translate-fe = {
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "translatefe";

      EnvironmentFile = [ config.sops.secrets."translate-fe/env".path ];

      ExecStart = "${translatefeExecstart}/bin/translate-fe";
    };
  };
}

If you are configuring a NixOS module (instead of declaring a custom service), the option might not always be called EnvironmentFile. For example, for the oauth2-proxy service, you would need to configure the services.oauth2-proxy.keyFile option:

  services.oauth2-proxy = {
    keyFile = config.sops.secrets."oauth2-proxy/env".path;
    enable = true;
    # …
  };

Usage Example: systemd credentials

In the previous examples, we configured the owner of each secret to the user account under which the service is running. But what if there is no such user account, because the service use systemd’s DynamicUser feature?

We can use systemd’s LoadCredential feature! For example, I supply the SMTP password to my Prometheus Alertmanager as follows:

{
  sops.secrets."alertmanager/smtp_pw" = {
    restartUnits = [ "alertmanager.service" ];
  };

  systemd.services.alertmanager.serviceConfig.LoadCredential = [
    "smtp_pw:${config.sops.secrets."alertmanager/smtp_pw".path}"
  ];

  services.prometheus.alertmanager = {
    enable = true;

    configuration = {
      global = {
        smtp_smarthost = "smtp.gmail.com:587";
        smtp_from = "alerts@example.net";
        smtp_auth_username = "alerts@example.net";
        smtp_auth_password_file = "/run/credentials/alertmanager.service/smtp_pw";
      };

      # …remaining config goes here…
    };
  };
}

Usage Example: samba users/passwords

In my blog post “Migrating my NAS from CoreOS/Flatcar Linux to NixOS”, I describe how to configure samba users and passwords (from sops-managed secrets) with an ExecStartPre shell script (which is very similar to the techniques already explained).

Conclusion

Managing secrets as separately-encrypted files in your config repository makes sense to me!

age’s ability to work with SSH keys makes for a really convenient setup, in my opinion. Encrypting secrets for the destination system’s SSH host key feels very elegant.

I hope the examples above are sufficient for you to efficiently configure secrets in NixOS!

Development shells with Nix: four quick examples

2025-07-27T08:50:00+02:00

I wanted to use GoCV for one of my projects (to find and extract paper documents from within a larger scan), without permanently having OpenCV on my system.

This seemed like a good example use-case to demonstrate a couple of Nix commands I like to use, covering quick interactive one-off dev shells to fully declarative, hermetic, reproducible, shareable dev shells.

Notably, you don’t need to use NixOS to run these commands! You can install and use Nix on any Linux system like Debian, Arch, etc., as long as you set a Nix path or use Flakes (see setup).

For comparison: The Debian Way

Before we start looking at Nix, I will show how to get GoCV running on Debian.

Let’s create a minimal Go program which uses a GoCV function like gocv.NewMat(), just to verify that we can compile this program:

package main

import "gocv.io/x/gocv"

func main() {
  gocv.NewMat()
}

If we try to build this on a Debian system, we get:

debian % mkdir -p /tmp/minimal
debian % cd /tmp/minimal

debian % cat > minimal.go <<'EOT'
package main
import "gocv.io/x/gocv"
func main() { gocv.NewMat(); }
EOT

debian % go mod init minimal
go: creating new go.mod: module minimal
go: to add module requirements and sums:
	go mod tidy

debian % go mod tidy
go: finding module for package gocv.io/x/gocv
go: downloading gocv.io/x/gocv v0.41.0
go: found gocv.io/x/gocv in gocv.io/x/gocv v0.41.0

debian % go build
# gocv.io/x/gocv
# [pkg-config --cflags  -- opencv4]
Package opencv4 was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv4.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv4', required by 'virtual:world', not found

On Debian, we can install OpenCV as follows:

debian % sudo apt install libopencv-dev

[…]

Summary:
  Upgrading: 7, Installing: 512, Removing: 0, Not Upgrading: 27
  Download size: 367 MB
  Space needed: 1590 MB / 281 GB available

Continue? [Y/n]

Saying “yes” to this prompt downloads and installs over 500 packages (takes a few minutes).

Now the build works:

debian % go build
debian % file minimal
minimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), […]

…but we have over 500 extra packages on our system that will now need to be updated in all eternity, therefore I would like to separate this one-off experiment from my usual system.

We could use Docker to start a Debian container and work inside that container, but, depending on the task, this can be cumbersome precisely because it’s a separate environment. For this example, I would need to specify a volume mount to make my input files available to the Docker container, and I would need to set up environment variables before programs inside the Docker container can open graphical windows on the host…

Let’s look at how we can use Nix to help us with that!

Setup: Nix-on-Debian (or Nix-on-Arch, or…)

Users of NixOS can skip this section, as NixOS systems include a ready-to-use Nix.

Before you can try the examples on your own computer, you need to complete these three steps:

Install Nix
Enable Flakes
Set a Nix path

Step 1: Install Nix

Users of Debian, Arch, Fedora, or other Linux systems first need to install Nix. Luckily, Nix is available for many popular Linux distributions:

Debian ships nix-setup-systemd
Arch Linux packages nix and provides documentation on the Nix Arch Wiki page. In practice, I installed the package and configured a couple of nixbld users.
More generally, there are Nix builds (rpm, deb, pacman) available for a number of distributions: https://github.com/nix-community/nix-installers

Step 2: Enable Flakes

Nix flakes are “a generic way to package Nix artifacts”.

Examples 3 and 4 use Nix flakes to pin dependencies, so we need to enable Nix flakes.

Step 3: Set a Nix path

For example 1 and 2, we want to use the Nix expression import .

On NixOS, this expression will follow the system version, meaning if you use import on a NixOS 25.05 installation, that will reference nixpkgs in version nixos-25.05.

On other Linux systems, you’ll see an error message like this:

debian-server % nix-shell -p pkg-config opencv
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

       at «string»:1:25:

            1| {...}@args: with import  args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (pkg-config) (opencv) ]; } ""
             |                         ^
(use '--show-trace' to show detailed location information)

We need to tell Nix which version of nixpkgs to use by setting the Nix search path:

debian-server % export NIX_PATH=nixpkgs=channel:nixos-25.05
debian-server % nix-shell -p pkg-config opencv
[nix-shell:/tmp/opencv]#

Alright! Now we are set up. Let’s jump into the first example!

Example 1: Interactive one-offs: nix-shell

Nix provides a middle-ground between installing OpenCV on your system (apt install like in the example above) and installing OpenCV in a separate Docker container: Nix can make available OpenCV without permanently installing it.

We can run nix-shell(1) to start a bash shell in which the specified packages are available. To successfully build Go code that uses GoCV, we need to have OpenCV available:

% nix-shell -p pkg-config opencv
these 194 paths will be fetched (175.80 MiB download, 764.10 MiB unpacked):
  /nix/store/ig2nk0hsha9xaailhaj69yv677nv95q4-abseil-cpp-20210324.2
  /nix/store/yw5xqn8lqinrifm9ij80nrmf0i6fdcbx-alsa-lib-1.2.13
[…]

[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

In case you were wondering: Yes, we do need to specify pkg-config in this nix-shell command explicitly, otherwise running pkg-config will run the host version (outside the dev shell), which cannot find opencv4.pc.

Example 2: nix-shell config file: shell.nix

Once we have a combination of packages that work for our project (in our example, just pkg-config and opencv), we can create a shell.nix (in any directory, but usually in the root of a project) which nix-shell (without the -p flag) will read:

{
  pkgs ? import  { },
}:
pkgs.mkShell {
  packages = with pkgs; [
    # Explicitly list pkg-config so that mkShell will arrange
    # for the PKG_CONFIG_PATH to find the .pc files.
    pkg-config
    opencv
  ];
}

…and then, we just run nix-shell:

% nix-shell
[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

If you’re curious, here are a couple of documentation pointers regarding the boilerplate around the list of packages:

Line 1 to 3 declare a function with an argument set — this is the required structure for nix-shell to be able to call your shell.nix file.
pkgs.mkShell is a convenience helper for use with nix-shell.
The with pkgs; part allows us to write opencv instead of pkgs.opencv.

By the way: With the nixd language server, editors with LSP support can show the versions that packages resolve to, point out your spelling mistakes, or provide features like “jump to definition”.

For example, in this screenshot, I was editing shell.nix in Emacs and was curious how the Nix source of the opencv package looked like. By pressing M-. (xref-find-definitions) with “point” over opencv, I got to opencv/4.x.nix in my local Nix store:

Example 3: Hermetic, pinned devShells: Nix Flakes

The previous examples used nixpkgs from your system (or Nix path), which means you don’t need to change the .nix file when you upgrade your system — depending on the use-case, I see this behavior as either convenient or terrifying.

For use-cases where it is important that the .nix file is built exactly the same way, no matter what version the surrounding OS uses, we can use Nix Flakes to build in a hermetic way, with dependency versions pinned in the flake.lock file.

A flake.nix contains the same mkShell expression as above, but declares structure around it: The mkShell expression goes into the outputs.devShells.x86_64-linux.default attribute and the inputs attribute contains Flake references that are available to this build:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";

  outputs =
    { self, nixpkgs }:
    {
      devShells.x86_64-linux.default =
        let
          pkgs = nixpkgs.legacyPackages.x86_64-linux;
        in
        pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
    };
}

By the way: Despite the name, it is a best practice to use nixpkgs.legacyPackages, which conceptually provides a single import nixpkgs result (for efficiency).

Now, I can use nix develop to get a shell with OpenCV:

% nix develop
michael@midna$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

The first nix develop run creates a flake.lock file, so running nix develop later will get us exactly the same environment. To update to newer versions, use nix flake update.

Tip: Instead of a shell, nix develop --command=emacs is also a useful variant.

Example 4: Making the Flake system-independent

Unfortunately, the above flake.nix hard-codes x86_64-linux, so it will not be usable on, say, an aarch64-linux (ARM) computer, or on a x86_64-darwin (Mac).

Having to explicitly specify the system by default is a long-standing criticism of Nix Flakes.

There are a number of workarounds. For example, we can use numtide/flake-utils and refactor our flake.nix to use its eachDefaultSystem convenience function:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        formatter = pkgs.nixfmt-tree;
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
      }
    );
}

Or we could use numtide/blueprint, its spiritual successor.

LucPerkins’s dev-templates have effectively inlined a version of this technique.

For a solution that isn’t part of Nix, but Nix-adjacent: devenv is a separate tool that is built on Nix (no longer using the CppNix implementation, but tvix actually), but with its own .nix files.

Tip: Keeping packages around

If you notice that nix develop or similar commands fetch packages despite the flake.lock not having changed, you can install the Flake into your profile to declare it as a gcroot to Nix:

% nix profile install .#devShells.x86_64-linux.default

But wait, isn’t that getting us into the same state as with The Debian Way? No! While OpenCV will remain available indefinitely if you install the flake into your profile, there still is a layer of separation: Within your system, OpenCV isn’t available, only when you start a development shell with nix-shell or nix develop.

Conclusion

How do the four examples above compare? Here’s an overview:

Example	Boilerplate	Pinned?	System-dependent?
Ex 1: `nix-shell -p …`	😊	no	no
Ex 2: `shell.nix`	🙂	no	no
Ex 3: `flake.nix`	😲	yes	yes
Ex 4: system-independent `flake.nix`	🤨	yes	no

For personal one-off experiments, I use nix-shell.

Once the experiment works, I typically want to pin the dependencies, so I use a flake.nix.

If this is software that isn’t just versioned, but also published (or worked on with multiple people/systems), I go through the effort of making it a system-independent flake.nix.

I hope in the future, it will become easier to write a system-independent flake.

Despite the rough edges, I appreciate the reproducibility and control that Nix gives me!

Migrating my NAS from CoreOS/Flatcar Linux to NixOS

2025-07-13T08:17:00+02:00

In this article, I want to show how to migrate an existing Linux server to NixOS — in my case the CoreOS/Flatcar Linux installation on my Network Attached Storage (NAS) PC.

I will show in detail how the previous CoreOS setup looked like (lots of systemd units starting Docker containers), how I migrated it into an intermediate state (using Docker on NixOS) just to get things going, and finally how I migrated all units from Docker to native NixOS modules step-by-step.

If you haven’t heard of NixOS, I recommend you read the first page of the NixOS website to understand what NixOS is and what sort of things it makes possible.

The target audience of this blog post is people interested in trying out NixOS for the use-case of a NAS, who like seeing examples to understand how to configure a system.

You can apply these examples by first following my blog post “How I like to install NixOS (declaratively)”, then making your way through the sections that interest you. If you prefer seeing the full configuration, skip to the conclusion.

Context/History

Over the last decade, I used a number of different operating systems for my NAS needs. Here’s an overview of the 2 NAS systems storage2 and storage3:

Year	storage2	storage3	Details (blog post)
2013	Debian on qnap	Debian on qnap	Wake-On-LAN with Debian on a qnap TS-119P2+
2016	CoreOS on PC	CoreOS on PC	Gigabit NAS (running CoreOS)
2023	CoreOS on PC	Ubuntu+ZFS on PC	My all-flash ZFS NAS build
2025	NixOS on PC	Ubuntu+ZFS on PC	→ you are here ←
?	NixOS on PC	NixOS+ZFS on PC	Converting more PCs to NixOS seems inevitable ;)

My NAS Software Requirements

(This post is only about software! For my usage patterns and requirements regarding hardware selection, see “Design Goals” in my My all-flash ZFS NAS build post (2023).)
Remote management: I really like the model of having the configuration of my network storage builds version-controlled and managed on my main PC. It’s a nice property that I can regain access to my backup setup by re-installing my NAS from my PC within minutes.
Automated updates, with easy rollback: Updating all my installations manually is not my idea of a good time. Hence, automated updates are a must — but when the update breaks, a quick and easy path to recovery is also a must.
- CoreOS/Flatcar achieved that with an A/B updating scheme (update failed? boot the old partition), whereas NixOS achieves that with its concept of a “generation” (update failed? select the old generation), which is finer-grained.

Why migrate from CoreOS/Flatcar to NixOS?

When I started using CoreOS, Docker was pretty new technology. I liked that using Docker containers allowed you to treat services uniformly — ultimately, they all expose a port of some sort (speaking HTTP, or Postgres, or…), so you got the flexibility to run much more recent versions of software on a stable OS, or older versions in case an update broke something.

Over a decade later, Docker is established tech. People nowadays take for granted the various benefits of the container approach.

So, here’s my list of reasons why I wasn’t satisfied with Flatcar Linux anymore.

R1. cloud-init is deprecated

The CoreOS cloud-init project was deprecated at some point in favor of Ignition, which is clearly more powerful, but also more cumbersome to get started with as a hobbyist. As far as I can tell, I must host my config at some URL that I then provide via a kernel parameter. The old way of just copying a file seems to no longer be supported.

Ignition also seems less convenient in other ways: YAML is no longer supported, only JSON, which I don’t enjoy writing by hand. Also, the format seems to change quite a bit.

As a result, I never made the jump from cloud-init to Ignition, and it’s not good to be reliant on a long-deprecated way to use your OS of choice.

R2. Container Bitrot

At some point, I did an audit of all my containers on the Docker Hub and noticed that most of them were quite outdated. For a while, Docker Hub offered automated builds based on a Dockerfile obtained from GitHub. However, automated builds now require a subscription, and I will not accept a subscription just to use my own computers.

R3. Dependency on a central service

If Docker at some point ceases operation of the Docker Hub, I am unable to deploy software to my NAS. This isn’t a very hypothetical concern: In 2023, Docker Hub announced the end of organizations on the Free tier and then backpedaled after community backlash.

Who knows how long they can still provide free services to hobbyists like myself.

R4. Could not try Immich on Flatcar

The final nail in the coffin was when I noticed that I could not try Immich on my NAS system! Modern web applications like Immich need multiple Docker containers (for Postgres, Redis, etc.) and hence only offer Docker Compose as a supported way of installation.

Unfortunately, Flatcar does not include Docker Compose.

I was not in the mood to re-package Immich for non-Docker-Compose systems on an ongoing basis, so I decided that a system on which I can neither run software like Immich directly, nor even run Docker Compose, is not sufficient for my needs anymore.

Reason Summary

With all of the above reasons, I would have had to set up automated container builds, run my own central registry and would still be unable to run well-known Open Source software like Immich.

Instead, I decided to try NixOS again (after a 10 year break) because it seems like the most popular declarative solution nowadays, with a large community and large selection of packages.

How does NixOS compare for my situation?

Same: I also need to set up an automated job to update my NixOS systems.
- I already have such a job for updating my gokrazy devices.
- Docker push is asynchronous: After a successful push, I still need extra automation for pulling the updated containers on the target host and restarting the affected services, whereas NixOS includes all of that.
Better: There is no central registry. With NixOS, I can push the build result directly to the target host via SSH.
Better: The corpus of available software in NixOS is much larger (including Immich, for example) and the NixOS modules generally seem to be expressed at a higher level of abstraction than individual Docker containers, meaning you can configure more features with fewer lines of config.

Prototyping in a VM

My NAS setup needs to work every day, so I wanted to prototype my desired configuration in a VM before making changes to my system. This is not only safer, it also allows me to discover any roadblocks, and what working with NixOS feels like without making any commitments.

I copied my NixOS configuration from a previous test installation (see “How I like to install NixOS (declaratively)”) and used the following command to build a VM image and start it in QEMU:

nix build .#nixosConfigurations.storage2.config.system.build.vm

export QEMU_NET_OPTS=hostfwd=tcp::2222-:22
export QEMU_KERNEL_PARAMS=console=ttyS0
./result/bin/run-nixplay-vm

The configuration instructions below can be tried out in this VM, and once you’re happy enough with what you have, you can repeat the steps on the actual machine to migrate.

Migrating

For the migration of my actual system, I defined the following milestones that should be achievable within a typical session of about an hour (after prototyping them in a VM):

M1. Install NixOS
M2. Set up remote disk unlock
M3. Set up Samba for access
M4. Set up SSH/rsync for backups
Everything extra is nice-to-have and could be deferred to a future session on another day.

In practice, this worked out exactly as planned: the actual installation of NixOS and setting up my config to milestone M4 took a little over one hour. All the other nice-to-haves were done over the following days and weeks as time permitted.

Tip: After losing data due to an installer bug in the 2000s, I have adopted the habit of physically disconnecting all data disks (= pulling out the SATA cable) when re-installing the system disk.

M1. Install NixOS

After following “How I like to install NixOS (declaratively)”, this is my initial configuration.nix:

{ modulesPath, lib, pkgs, ... }:

{
  imports =
    [
      (modulesPath + "/installer/scan/not-detected.nix")
      ./hardware-configuration.nix
      ./disk-config.nix
    ];

  # Adding michael as trusted user means
  # we can upgrade the system via SSH (see Makefile).
  nix.settings.trusted-users = [ "michael" "root" ];
  # Clean the Nix store every week.
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 7d";
  };

  boot.loader.systemd-boot = {
    enable = true;
    configurationLimit = 10;
  };
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "storage2";
  time.timeZone = "Europe/Zurich";

  # Use systemd for networking
  services.resolved.enable = true;
  networking.useDHCP = false;
  systemd.network.enable = true;

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
  };

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  users.mutableUsers = false;
  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "networkmanager" "wheel" ];
    initialPassword = "secret";  # XXX: change!
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
  ];

  programs.zsh.enable = true;

  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

All following sections describe changes within this configuration.nix.

All devices in my home network obtain their IP address via DHCP. If I want to make an IP address static, I configure it accordingly on my router.

My NAS PCs have one specialty with regards to IP addressing: They are reachable via IPv4 and IPv6, and the IPv6 address can be derived from the IPv4 address.

Hence, I changed the systemd-networkd configuration from above such that it configures a static IPv6 address in a dynamically configured IPv6 network:

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
    ipv6AcceptRAConfig = {
      Token = "::10:0:0:252";
    };
  };

✅ This fulfills milestone M1.

M2. Set up remote disk unlock

To unlock my encrypted disks on boot, I have a custom systemd service unit that uses wget(1) and cryptsetup(8) to split the key file between the NAS and a remote server (= an attacker needs both pieces to unlock).

With CoreOS/Flatcar, my cloud-init configuration looked as follows:

coreos:
  units:
    - name: unlock.service
      command: start
      content: |
        [Unit]
        Description=unlock hard drive
        Wants=network.target
        After=systemd-networkd-wait-online.service
        Before=samba.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # Wait until the host is actually reachable.
        ExecStart=/bin/sh -c "c=0; while [ $c -lt 5 ]; do /bin/ping6 -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"
        ExecStart=/bin/sh -c "vgchange -ay"
        ExecStart=/bin/mount /dev/mapper/data-data /srv

write_files:
  - path: /etc/ssl/certs/r.zekjur.net.crt
    content: |
      -----BEGIN CERTIFICATE-----
      MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
      […]
      -----END CERTIFICATE-----

I converted it into the following NixOS configuration:

  systemd.services.unlock = {
    wantedBy = [ "multi-user.target" ];
    description = "unlock hard drive";
    wants = [ "network.target" ];
    after = [ "systemd-networkd-wait-online.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = "yes";
      ExecStart = [
        # Wait until the host is actually reachable.
        ''/bin/sh -c "c=0; while [ $c -lt 5 ]; do ${pkgs.iputils}/bin/ping -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdb2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdc2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"''

        ''/bin/sh -c "${pkgs.lvm2}/bin/vgchange -ay"''
        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
      ];
    };

  };

We’ll also need to store the custom TLS certificate file on disk. For that, we can use the environment. configuration:

  environment.etc."ssl/certs/r.zekjur.net.crt".text = ''
-----BEGIN CERTIFICATE-----
MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
[…]
-----END CERTIFICATE-----
'';

The references like ${pkgs.wget} will be replaced with a path to the Nix store (→ nix.dev documentation). On CoreOS/Flatcar, I was limited to using just the (minimal set of) software included in the base image, or I had to reach for Docker. On NixOS, we can use all packages available in nixpkgs.

After deploying and rebooting, I can access my unlocked disk under /srv! 🎉

% df -h /srv
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/data-data   15T   14T  342G  98% /srv

When listing my files, I noticed that the group id was different between my old system and the new system. This can be fixed by explicitly specifying the desired group id:

  users.groups.michael = {
    gid = 1000;  # for consistency with storage3
  };

✅ M2 is complete.

M3. Set up Samba for access

Whereas I want to configure remote disk unlock at the systemd service level, for Samba I want to use Docker: I wanted to first transfer my old (working) Docker-based setups as they are, and only later convert them to Nix.

We enable the Docker NixOS module which sets up the daemons that Docker needs and whatever else is needed to make it work:

  virtualisation.docker.enable = true;

This is already sufficient for other services to use Docker, but I also want to be able to run the docker command interactively for debugging. Therefore, I added docker to systemPackages:

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    docker
  ];

After deploying this configuration, I can run docker run -ti debian to verify things work.

The cloud-init version of samba looked like this:

coreos:
  units:
    - name: samba.service
      command: start
      content: |
        [Unit]
        Description=samba server
        After=docker.service unlock.mount
        Requires=docker.service unlock.mount

        [Service]
        Restart=always
        StartLimitInterval=0

        # Always pull the latest version (bleeding edge).
        ExecStartPre=-/usr/bin/docker pull stapelberg/docker-samba:latest

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ExecStartPre=-/usr/bin/docker kill smb
        ExecStartPre=-/usr/bin/docker rm smb
        ExecStartPre=-/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf'
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared
        ExecStartPre=/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared

        ExecStart=/usr/bin/docker run \
          -p 137:137 \
          -p 138:138 \
          -p 139:139 \
          -p 445:445 \
          --tmpfs=/run \
          -v /srv/data:/srv/data \
          --name smb \
          -t \
          smb-prepared \
            /usr/sbin/smbd --foreground --debug-stdout --no-process-group

We can translate this 1:1 to NixOS:

  systemd.services.samba = {
    wantedBy = [ "multi-user.target" ];
    description = "samba server";
    after = [ "unlock.service" ];
    requires = [ "unlock.service" ];
    serviceConfig = {
      Restart = "always";
      StartLimitInterval = 0;
      ExecStartPre = [
        # Always pull the latest version.
        ''-${pkgs.docker}/bin/docker pull stapelberg/docker-samba:latest''

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ''-${pkgs.docker}/bin/docker kill smb''
        ''-${pkgs.docker}/bin/docker rm smb''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf' ''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
      ];

      ExecStart = ''-${pkgs.docker}/bin/docker run \
           -p 137:137 \
           -p 138:138 \
           -p 139:139 \
           -p 445:445 \
           --tmpfs=/run \
           -v /srv/data:/srv/data \
           --name smb \
           -t \
           smb-prepared \
             /usr/sbin/smbd --foreground --debug-stdout --no-process-group
             '';
    };
  };
}

✅ Now I can manage my files over the network, which completes M3!

M4. Set up SSH/rsync for backups

For backing up data, I use rsync over SSH. I restrict this SSH access to run only rsync commands by using rrsync (in a Docker container). To configure the SSH authorized_keys(5) , we set:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  };

✅ A successful test backup run completes milestone M4!

Nice-to-haves

N1. Prometheus Node Exporter

I like to monitor all my machines with Prometheus (and Grafana). For network connectivity and authentication, I use the Tailscale mesh VPN.

To install Tailscale, I enable its NixOS module and make the tailscale command available:

  services.tailscale.enable = true;
  environment.systemPackages = with pkgs; [ tailscale ];

After deploying, I run sudo tailscale up and open the login link in my browser.

The Prometheus Node Exporter can also easily be enabled through its NixOS module:

  services.prometheus.exporters.node = {
    enable = true;
    listenAddress = "storage2.example.ts.net";
  };

However, this isn’t reliable yet: When Tailscale’s startup takes a while during system boot, the Node Exporter might burn through its entire restart budget when it cannot listen on the Tailscale IP address yet. We can enable indefinite restarts for the service to eventually come up:

  systemd.services."prometheus-node-exporter" = {
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      RestartSec = 1;
    };
  };

N2. Reliable mounting

While migrating my setup, I noticed that calling mount(8) from unlock.service directly is not reliable, and it’s better to let systemd manage the mounting:

  fileSystems."/srv" = {
    device = "/dev/mapper/data-data";
    fsType = "ext4";
    options = [
      "nofail"
      "x-systemd.requires=unlock.service"
    ];
  };

Afterwards, I could just remove the mount(8) call from unlock.service:

@@ -247,7 +247,10 @@ fry/U6A=
         ''/bin/sh -c "${pkgs.lvm2.bin}/bin/vgchange -ay"''
-        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
+        # Let systemd mount /srv based on the fileSystems./srv
+        # declaration to prevent race conditions: mount
+        # might not succeed while the fsck is still in progress,
+        # for example, which otherwise makes unlock.service fail.
       ];
     };

In systemd services, I can now depend on the /srv mount unit:

  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N3. nginx-healthz

To save power, I turn off my NAS when they are not in use.

My backup orchestration uses Wake-on-LAN to start a wakeup and needs to wait until the NAS is fully booted up and has mounted its /srv mount before it can start backup jobs.

For this purpose, I have configured a web server (without any files) that depends on the /srv mount. So, once the web server responds to HTTP requests, we know /srv is mounted.

The cloud-init config looked as follows:

coreos:
  units:
    - name: healthz.service
      command: start
      content: |
        [Unit]
        Description=nginx for /srv health check
        Wants=network.target
        After=srv.mount
        Requires=srv.mount
        StartLimitInterval=0

        [Service]
        Restart=always
        ExecStartPre=/bin/sh -c 'systemctl is-active docker.service'
        ExecStartPre=/usr/bin/docker pull nginx:1
        ExecStartPre=-/usr/bin/docker kill nginx-healthz
        ExecStartPre=-/usr/bin/docker rm -f nginx-healthz
        ExecStart=/usr/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1

The Docker version (ported from Flatcar Linux) looks like this:

  systemd.services.healthz = {
    description = "nginx for /srv health check";
    wants = [ "network.target" ];
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''/bin/sh -c 'systemctl is-active docker.service' ''
        ''-${pkgs.docker}/bin/docker pull nginx:1''
        ''-${pkgs.docker}/bin/docker kill nginx-healthz''
        ''-${pkgs.docker}/bin/docker rm -f nginx-healthz''
      ];

      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1
        ''
      ];
    };
  };

This configuration gets a lot simpler when migrating it from Docker to NixOS:

  # Signal readiness on HTTP port 8200 once /srv is mounted:
  networking.firewall.allowedTCPPorts = [ 8200 ];
  services.caddy = {
    enable = true;
    virtualHosts."http://10.0.0.252:8200".extraConfig = ''
      respond "ok"
    '';
  };
  systemd.services.caddy = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N4. NixOS Jellyfin

The Docker version (ported from Flatcar Linux) looks like this:

  networking.firewall.allowedTCPPorts = [ 4414 8096 ];
  systemd.services.jellyfin = {
    wantedBy = [ "multi-user.target" ];
    description = "jellyfin";
    after = [ "docker.service" "srv.mount" ];
    requires = [ "docker.service" "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''-${pkgs.docker}/bin/docker pull lscr.io/linuxserver/jellyfin:latest''
        ''-${pkgs.docker}/bin/docker rm jellyfin''
      ];
      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
          --rm \
          --net=host \
          --name=jellyfin \
          -e TZ=Europe/Zurich \
          -v /srv/jellyfin/config:/config \
          -v /srv/data/movies:/data/movies:ro \
          -v /srv/data/series:/data/series:ro \
          -v /srv/data/mp3:/data/mp3:ro \
          lscr.io/linuxserver/jellyfin:latest
        ''
      ];
    };
  };

As before, when using jellyfin from NixOS, the configuration gets simpler:

  services.jellyfin = {
    enable = true;
    openFirewall = true;
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

For a while, I had also set up compatibility symlinks that map the old location (/data/movies, inside the Docker container) to the new location (/srv/data/movies), but I encountered strange issues in Jellyfin and ended up just re-initializing my whole Jellyfin state. While the required configuration had more lines, I found it neat to move it into its own file, so here is how to do that:

Remove the lines above from configuration.nix and move them into jellyfin.nix:

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}:

{
  services.jellyfin = {
    enable = true;
    openFirewall = true;
    dataDir = "/srv/jellyfin";
    cacheDir = "/srv/jellyfin/config/cache";
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };
}

Then, in configuration.nix, add jellyfin.nix to the imports:

   imports = [
     ./hardware-configuration.nix
     ./jellyfin.nix
   ];

N5. NixOS samba

To use Samba from NixOS, I replaced my systemd.services.samba config from M3 with this:

  services.samba = {
    enable = true;
    openFirewall = true;
    settings = {
      "global" = {
        "map to guest" = "bad user";
      };
      "data" = {
        "path" = "/srv/data";
        "comment" = "public data";
        "read only" = "no";
        "create mask" = "0775";
        "directory mask" = "0775";
        "guest ok" = "yes";
      };
    };
  };
  system.activationScripts.samba_user_create = ''
      smb_password="secret"
      echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s michael
    '';

Note: Setting the samba password in the activation script works for small setups, but if you want to keep your samba passwords out of the Nix store, you’ll need to use a different approach. On a different machine, I use sops-nix to manage secrets and found that refactoring the smbpasswd call like so works reliably:

let
  setPasswords = pkgs.writeShellScript "samba-set-passwords" ''
    set -euo pipefail
    for user in michael; do
        smb_password="$(cat /run/secrets/samba_passwords/$user)"
        echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s $user
    done
  '';
in
 {
  # …
  services.samba = {
    # …as above…
  }

  systemd.services.samba-smbd.serviceConfig.ExecStartPre = [
    "${setPasswords}"
  ];

  sops.secrets."samba_passwords/michael" = {
    restartUnits = [ "samba-smbd.service" ];
  };
}

I also noticed that NixOS does not create a group for each user by default, but I am used to managing my permissions like that. We can easily declare a group like so:

  users.groups.michael = {
    gid = 1000; # for consistency with storage3
  };
  users.users.michael = {
    extraGroups = [
      "wheel" # Enable ‘sudo’ for the user.
      "docker"
      # By default, NixOS does not add users to their own group:
      # https://github.com/NixOS/nixpkgs/issues/198296
      "michael"
    ];
  };

N6. NixOS rrsync

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

To use rrsync from NixOS, I changed the configuration like so:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.rrsync}/bin/rrsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

N7. sync.pl script

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/data:/srv/data -v /root/.ssh:/root/.ssh:ro -v /etc/ssh:/etc/ssh:ro -v /etc/static/ssh:/etc/static/ssh:ro -v /nix/store:/nix/store:ro stapelberg/docker-sync",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

I wanted to stop managing the following Dockerfile to ship sync.pl:

FROM debian:stable

# Install full perl for Data::Dumper
RUN apt-get update \
    && apt-get install -y rsync ssh perl

ADD sync.pl /usr/bin/

ENTRYPOINT ["/usr/bin/sync.pl"]

To get rid of the Docker container, I translated the sync.pl file into a Nix expression that writes the sync.pl Perl script to the Nix store:

{ pkgs }:

# For string literal escaping rules (''${), see:
# https://nix.dev/manual/nix/2.26/language/string-literals#string-literals

# For writers.writePerlBin, see https://wiki.nixos.org/wiki/Nix-writers

pkgs.writers.writePerlBin "syncpl" { libraries = []; } ''
# This script is run via ssh from dornröschen.
use strict;
use warnings;
use Data::Dumper;

if (my ($destination) = ($ENV{SSH_ORIGINAL_COMMAND} =~ /^([a-z0-9.]+)$/)) {
    print STDERR "rsync version: " . `${pkgs.rsync}/bin/rsync --version` . "\n\n";
    my @rsync = (
        "${pkgs.rsync}/bin/rsync",
        "-e",
        "ssh",
        "--max-delete=-1",
        "--verbose",
        "--stats",
        # Intentionally not setting -X for my data sync,
        # where there are no full system backups; mostly media files.
        "-ax",
        "--ignore-existing",
        "--omit-dir-times",
        "/srv/data/",
        "''${destination}:/",
    );
    print STDERR "running: " . Dumper(\@rsync) . "\n";
    exec @rsync;
} else {
    print STDERR "Could not parse SSH_ORIGINAL_COMMAND.\n";
}
''

I can then reference this file by importing it in my configuration.nix and pointing it to the pkgs expression of my NixOS configuration:

{ modulesPath, lib, pkgs, ... }:

let
  syncpl = import ./syncpl.nix { pkgs = pkgs; };
in {
  imports = [ ./hardware-configuration.nix ];

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${syncpl}/bin/syncpl",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

  # For interactive usage (when debugging):
  environment.systemPackages = [ syncpl ];

  # …
}

This works, but is it the best approach? Here are some thoughts:

By managing this script in a Nix expression, I can no longer use my editor’s Perl support.
- I could probably also keep sync.pl as a separate file and use string interpolation in my Nix expression to inject an absolute path to the rsync binary into the script.
Another alternative would be to add a wrapper script to my Nix expression which ensures that $PATH contains rsync and then the script wouldn’t need an absolute path anymore.
For small glue scripts like this one, I consider it easier to manage the contents “inline” in the Nix expression, because it means one fewer file in my config directory.

N8. Sharing configs

I want to configure all my NixOS systems such that my user settings are identical everywhere.

To achieve that, I can extract parts of my configuration.nix into a user-settings.nix and then declare an accompanying flake.nix that provides this expression as an output.

After publishing these files in a git repository, I can reference said repository in my flake.nix:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    stapelbergnix.url = "github:stapelberg/nix";
  };

  outputs =
    {
      self,
      nixpkgs,
	  stapelbergnix,
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.storage2 = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          ./configuration.nix
          stapelbergnix.lib.userSettings
          # Not on this machine; We have our own networking config:
          # stapelbergnix.lib.systemdNetwork
          # Use systemd-boot as bootloader
          stapelbergnix.lib.systemdBoot
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Everything declared in the user-settings.nix can now be removed from configuration.nix!

N9. Trying immich!

One of the motivating reasons for switching away from CoreOS/Flatcar was that I couldn’t try Immich, so let’s give it a shot on NixOS:

  services.immich = {
    enable = true;
    host = "10.0.0.252";
    port = 2283;
    openFirewall = true;
    mediaLocation = "/srv/immich";
  };

  # Because /srv is a separate file system, we need to declare:
  systemd.services."immich-server" = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

Conclusion

You can find the full configuration directory on GitHub.

I am pretty happy with this NixOS setup! Previously (with CoreOS/Flatcar), I could declaratively manage my base system, but had to manage tons of Docker containers in addition. With NixOS, I can declaratively manage everything (or as much as makes sense).

Custom configuration like my SSH+rsync-based backup infrastructure can be expressed cleanly, in one place, and structured at the desired level of abstraction/reuse.

If you’re considering managing at least one other system with NixOS, I would recommend it! One of my follow-up projects is to convert storage3 (my other NAS build) from Ubuntu Server to NixOS as well to cut down on manual management. Being able to just copy the entire config to set up another system, or try out an idea in a throwaway VM, is just such a nice workflow 🥰

…but if you have just a single system to manage, probably all of this is too complicated.

How I like to install NixOS (declaratively)

2025-06-01T08:20:43+02:00

Introduction: Declarative?

The term declarative means that you describe what should be accomplished, not how. For NixOS, that means you declare what software you want your system to include (add to config option environment.systemPackages, or enable a module) instead of, say, running apt install.

A nice property of the declarative approach is that your system follows your configuration, so by reverting a configuration change, you can cleanly revert the change to the system as well.

I like to manage declarative configuration files under version control, typically with Git.

When I originally set up my current network storage build, I chose CoreOS (later Flatcar Container Linux) because it was an auto-updating base system with a declarative cloud-init config.

Ways of installing NixOS

Graphical Installer: Only for Desktops

The NixOS manual’s “Installation” section describes a graphical installer (“for desktop users”, based on the Calamares system installer and added in 2022) and a manual installer.

With the graphical installer, it’s easy to install NixOS to disk: just confirm the defaults often enough and you’ll end up with a working system. But there are some downsides:

You need to manually enable SSH after the installation — locally, not via the network.
The graphical installer generates an initial NixOS configuration for you, but there is no way to inject your own initial NixOS configuration.

The graphical installer is clearly not meant for remote installation or automated installation.

Manual Installation

The manual installer on the other hand is too manual for my taste: expand “Example 2” and “Example 3” in the NixOS manual’s Installation summary section to get an impression. To be clear, the steps are very doable, but I don’t want to install a system this way in a hurry. For one, manual procedures are prone to mistakes under stress. And also, copy & pasting commands interactively is literally the opposite of writing declarative configuration files.

Network Installation: nixos-anywhere

Ideally, I would want to perform most of the installation from the comfort of my own PC, meaning the installer must be usable over the network. Also, I want the machine to come up with a working initial NixOS configuration immediately after installation (no manual steps!).

Luckily, there is a (community-provided) solution: nixos-anywhere. You take care of booting a NixOS installer, then run a single command and nixos-anywhere will SSH into that installer, partition your disk(s) and install NixOS to disk. Notably, nixos-anywhere is configured declaratively, so you can repeat this step any time.

(I know that nixos-anywhere can even SSH into arbitrary systems and kexec-reboot them into a NixOS installer, which is certainly a cool party trick, but I like the approach of explicitly booting an installer better as it seems less risky and more generally applicable/repeatable to me.)

Setup: Installing Nix

I want to use NixOS for one of my machines, but not (currently) on my main desktop PC.

Hence, I installed only the nix tool (for building, even without running NixOS) on Arch Linux:

% sudo pacman -S nix
% sudo groupadd -r nixbld
% for n in $(seq 1 24); do sudo useradd -c "Nix build user $n" \
    -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(which nologin)" \
    nixbld$n; done
% sudo systemctl enable --now nix-daemon.socket

Now, running nix-shell -p hello should drop you in a new shell in which the GNU hello package is installed:

% export NIX_PATH=nixpkgs=channel:nixos-25.05
% nix-shell -p hello
hello

[nix-shell:/tmp]$ hello
Hello, world!

By the way, the Nix page on the Arch Linux wiki explains how to use nix to install packages, but that’s not what I am interested in: I only want to remotely manage NixOS systems.

Building your own installer

Previously, I said “you take care of booting a NixOS installer”, and that’s easy enough: write the ISO image to a USB stick and boot your machine from it (or select the ISO and boot your VM).

But before we can log in remotely via SSH, we need to manually set a password. I also need to SSH with the TERM=xterm environment variable because the termcap file of rxvt-unicode (my preferred terminal) is not included in the default NixOS installer environment. Similarly, my configured locales do not work and my preferred shell (Zsh) is not available.

Wouldn’t it be much nicer if the installer was pre-configured with a convenient environment?

With other Linux distributions, like Debian, Fedora or Arch Linux, I wouldn’t attempt to re-build an official installer ISO image. I’m sure their processes and tooling work well, but I am also sure it’s one extra thing I would need to learn, debug and maintain.

But building a NixOS installer is very similar to configuring a regular NixOS system: same configuration, same build tool. The procedure is documented in the official NixOS wiki.

I copied the customizations I would typically put into configuration.nix, imported the installation-cd-minimal.nix module from nixpkgs and put the result in the iso.nix file:

{ config, pkgs, ... }:

{
  imports = [
    
    
  ];

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "wheel" ];
    initialPassword = "SGZ3odMZIesxTuh2Y2pUaJA";  # random for this post
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    rxvt-unicode  # for terminfo
    lshw
  ];

  programs.zsh.enable = true;
  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

To build the ISO image, I set the NIX_PATH environment variable to point nix-build(1) to the iso.nix file and to select the upstream channel for NixOS 25.05:

% export NIX_PATH=nixos-config=$PWD/iso.nix:nixpkgs=channel:nixos-25.05
% nix-build '' -A config.system.build.isoImage

After about 1.5 minutes on my 2025 high-end Linux PC, the installer ISO can be found in result/iso/nixos-minimal-25.05.802216.55d1f923c480-x86_64-linux.iso (1.46 GB in size in my case).

Enabling Nix Flakes

Unfortunately, the nix project has not yet managed to enable the “experimental” new command-line interface (CLI) by default yet, despite 5+ years of being available, so we need to create a config file and enable the modern nix-command interface:

% mkdir -p ~/.config/nix
% echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

How can you tell old from new? The old commands are hyphenated (nix-build), the new ones are separated by a blank space (nix build).

You’ll notice I also enabled Nix flakes, which I use so that my nix builds are hermetic and pinned to a certain revision of nixpkgs and any other nix modules I want to include in my build. I like to compare flakes to version lock file in other programming environments: the idea is that building the system in 5 months will yield the same result as it does today.

To verify that flakes work, run nix shell (not nix-shell):

% nix shell nixpkgs#hello
/tmp 2 % hello
Hello, world!

(Re-)Installation Steps

For reference, here is the configuration I use to create a new VM for NixOS in Proxmox. The most important setting is bios=ovmf (= UEFI boot, which is not the default), so that I can use the same boot loader configuration on physical machines as in VMs:

Before we can boot our (unsigned) installer, we need to enter the UEFI setup and disable Secure Boot. Note that Proxmox enables Secure Boot by default, for example.

Then, boot the custom installer ISO on the target system, and ensure ssh michael@nixos.lan works without prompting for a password.

Declare a flake.nix with the following content:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

    disko.url = "github:nix-community/disko";
    # Use the same version as nixpkgs
    disko.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs =
    {
      nixpkgs,
      disko,
      ...
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.zammadn = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          disko.nixosModules.disko
          ./configuration.nix
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Declare your disk config in disk-config.nix:

disk-config.nix

My 2025 high-end Linux PC 🐧

2025-05-15T15:44:24+02:00

Turns out my previous attempt at this build had a faulty CPU! With the CPU replaced, the machine now is stable and fast! 🚀 In this article, I’ll go into a lot more detail about the component selection, but in a nutshell, I picked an Intel 285K CPU for low idle power, chose a 4TB SSD so I don’t have to worry about running out of storage quickly, and a capable nvidia graphics card to drive my Dell UP3218K 8K monitor.

Components

Which components did I pick for this build? Here’s the full list:

Price	Type	Article
140 CHF	Case	Fractal Define 7 Compact Black Solid
155 CHF	Power Supply	Corsair RM850x
233 CHF	Mainboard	ASUS PRIME Z890-P
620 CHF	CPU	Intel Core Ultra 9 285K
120 CHF	CPU fan	Noctua NH-D15 G2
39 CHF	Case fan	Noctua NF-A14 PWM (140 mm)
209 CHF	RAM	64 GB DDR5-6400 Corsair Vengeance (2 x 32GB)
280 CHF	Disk	4000 GB Samsung 990 Pro
554 CHF	GPU	MSI GeForce RTX 3060 Ti GAMING X TRIO

Total: 2350 CHF

…and the next couple of sections go into detail on how I selected these components.

Case

I have been a fan of Fractal cases for a couple of generations. In particular, I realized that the “Compact” series offers plenty of space even for large graphics cards and CPU coolers, so that’s now my go-to case: the Fractal Define 7 Compact (Black Solid).

My general requirements for a PC case are as follows:

No extra effort should be required for the case to be as quiet as possible.
The case should not have any sharp corners (no danger of injury!).
The case should provide just enough space for easy access to your components.
The more support the case has to encourage clean cable routing, the better.
USB3 front panel headers should be included.

I really like building components into the case and working with the case. There are no sharp edges, the mechanisms are a pleasure to use and the cable-management is well thought-out.

The only thing that wasn’t top-notch is that Fractal ships the case screws in sealed plastic packages that you need to cut open. I would have wished for a re-sealable plastic baggie so that one can keep the unused screws instead of losing them.

With this build, I have standardized all my PCs into Fractal Define 7 Compact Black cases!

Power Supply

I wanted to keep my options open regarding upgrading to an nvidia 50xx series graphics card at a later point. Those models have a TGP (“Total Graphics Power”) of 575 watts, so I needed a power supply that delivers enough power for the whole system even at peak power usage in all dimensions.

I ended up selecting the Corsair RM850x, which reviews favorably (“leader in the 850W gold category”) and was available at my electronics store of choice.

This was a good choice: the PSU indeed runs quiet, and I really like the power cables (e.g. the GPU cable) that they include: they are very flexible, which makes them easy to cable-manage.

One interesting realization was that it’s more convenient to not use the PSU’s 12VHPWR cable, but instead stick to the older 8-pin power connectors for the GPU in combination with a 12VHPWR-to-8-pin adapter. The reason is that the 12VHPWR connector’s locking mechanism is very hard to unlock, so when swapping out the GPU (as I had to do a number of times while trouble-shooting), unlocking an 8-pin connector is much easier…

SSD disk

I have been avoiding PCIe 5 SSDs so far because they consume a lot more power compared to PCIe 4 SSDs. While bulk streaming data transfer rates are higher on PCIe 5 SSDs, random transfers are not significantly faster. Most of my compute workload are random transfers, not large bulk transfers.

The power draw situation with PCIe 5 SSDs seems to be getting better lately, with the Phison E31T being the first controller that implements power saving. A disk that uses the E31T controller is the Corsair Force Series MP700 Elite. Unfortunately, said disk was unavailable when I ordered.

Instead, I picked the Samsung 990 Pro with 4 TB. I have had good experiences with the Samsung Pro series over the years (never had one die or degrade performance), and my previous 2 TB disk was starting to fill up, so the extra storage space is appreciated.

Onboard 2.5GbE Network Card

One annoying realization is that most mainboard vendors seem to have moved to 2.5 GbE (= 2.5 Gbit/s ethernet) onboard network cards. I would have been perfectly happy to play it safe and buy another Intel I225 1 GbE network card, as long as it just works with Linux.

In the 2.5 GbE space, the main players seem to be Realtek and Intel. Most mainboard vendors opted for Realtek as far as I could see.

Linux includes the r8169 driver for Realtek network cards, but whether the card will work out of the box depends on the exact revision of the network card! For example:

The AsRock Z890 Pro-A has rev 8125B. lshw: firmware=rtl8125b-2_0.0.2 07/13/20
The ASUS PRIME Z890-P has rev 8125D. lshw: firmware=rtl8125d-1_0.0.7 10/15/24

For revision 8125D, you need a recent-enough Linux version (6.13+) that includes commit “r8169: add support for RTL8125D”, accompanied by a recent-enough linux-firmware package.

Even with the latest firmware, there is some concern around stability and ASPM support. See for example this ServerFault post by someone working on the r8169 driver. But, despite the Intel 1 GbE options being well-supported at this point, Intel’s 2.5 GbE options might not fare any better than the Realtek ones: I found reports of instability with Intel’s 2.5 GbE network cards.

That said, aside from the annoying firmware requirements, the Realtek 2.5 GbE card seems to work fine for me in practice.

Mainboard

Despite the suboptimal network card choice, I decided to stick to the ASUS PRIME series of mainboards, as I made good experiences with those in my past few builds. Here are a couple of thoughts on the ASUS PRIME Z890-P mainboard I went with:

I like the quick-release PCIe mechanism: ASUS understood that people had trouble unlocking large graphics cards from their PCIe slot, so they added a lever-like mechanism that is easily reachable. In my couple of usages, this worked pretty well!
I wrote about slow boot times with my 2022 PC build that were caused by time-consuming memory training. On this ASUS board, I noticed that the board blinks the Power LED to signal that memory training is in progress. Very nice! It hadn’t occurred to me previously that the various phases of the boot could be signaled by different Power LED blinking patterns :)
- The downside of this feature is: While the machine is in suspend-to-RAM, the Power LED also blinks! This is annoying, so I might just disconnect the Power LED entirely.
The UEFI firmware includes what they call a Q-Dashboard: An overview of what is installed/connected in which slot. Quite nice:

One surprising difference between the two mainboards I tested was that the AsRock Z890 Pro-A does not seem to report the correct DIMM clock in lshw, whereas the ASUS does:

--- lshw-intel-285k-asrock.txt	2025-04-30 20:35:24 +0200
+++ lshw-intel-285k-asus.txt		2025-04-30 21:39:52 +0200
      *-firmware
           description: BIOS
-          vendor: American Megatrends International, LLC.
+          vendor: American Megatrends Inc.
           physical id: 0
-          version: 2.25
-          date: 03/24/2025
+          version: 1601
+          date: 02/07/2025
           size: 64KiB
-          capacity: 32MiB
+          capacity: 16MiB
           capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd acpi biosbootspecification uefi
[…]
      *-memory
           description: System Memory
-          physical id: 9
+          physical id: e
           slot: System board or motherboard
           size: 64GiB
         *-bank:0
-             description: DIMM [empty]
+             description: [empty]
              physical id: 0
              slot: Controller0-ChannelA-DIMM0
         *-bank:1
-             description: DIMM Synchronous 4800 MHz (0,2 ns)
+             description: DIMM Synchronous 6400 MHz (0,2 ns)
              product: CMK64GX5M2B6400C32
              vendor: Corsair
              physical id: 1
@@ -40,13 +42,13 @@
              slot: Controller0-ChannelA-DIMM1
              size: 32GiB
              width: 64 bits
-             clock: 505MHz (2.0ns)
+             clock: 2105MHz (0.5ns)
         *-bank:2
-             description: DIMM [empty]
+             description: [empty]
              physical id: 2
              slot: Controller0-ChannelB-DIMM0
         *-bank:3
-             description: DIMM Synchronous 4800 MHz (0,2 ns)
+             description: DIMM Synchronous 6400 MHz (0,2 ns)
              product: CMK64GX5M2B6400C32
              vendor: Corsair
              physical id: 3
@@ -54,7 +56,7 @@
              slot: Controller0-ChannelB-DIMM1
              size: 32GiB
              width: 64 bits
-             clock: 505MHz (2.0ns)
+             clock: 2105MHz (0.5ns)
[…]

I haven’t checked if there are measurable performance differences (e.g. if the XMP profile is truly active), but at least you now know to not necessarily trust what lshw can show you.

CPU fan

I am a long-time fan of Noctua’s products: This company makes silent fans with great cooling capacity that work reliably! For many years, I have swapped out all the fans of each of my PCs with Noctua fans, and it was always an upgrade. Highly recommended.

Hence, it is no question that I picked the latest and greatest Noctua CPU cooler for this build: the Noctua NH-D15 G2. There are a couple of things to pay attention to with this cooler:

I decided to configure it with one fan instead of two fans: Using only one fan will be the quietest setup, yet still have plenty of cooling capacity for this setup.
There are 3 different versions that differ in how their base plate is shaped. Noctua recommends: “For LGA1851, we generally recommend the regular standard version with medium base convexity” (https://noctua.at/en/intel-lga1851-all-you-need-to-know)
With a height of 168 mm, this cooler fits well into the Fractal Define 7 Compact Black.

CPU and GPU: Idle Power vs. Peak Performance

CPU choice: Intel over AMD

Probably the point that raises most questions about this build is why I selected an Intel CPU over an AMD CPU. The primary reason is that Intel CPUs are so much better at power saving!

Let me explain: Most benchmarks online are for gamers and hence measure a usage curve that goes “start game, run PC at 100% resources for hours”. Of course, when you never let the machine idle, you would care about power efficiency: how much power do you need to use to achieve the desired result?

My use-case is software development, not gaming. My usage curve oscillates between “barely any usage because Michael is reading text” to “complete this compilation as quickly as possible with all the power available”. For me, I need both absolute power consumption at idle, and absolute performance to be best-of-class.

AMD’s CPUs offer great performance (the recently released Ryzen 9 9950X3D is even faster than the Intel 9 285K), and have great power efficiency, but poor power consumption at idle: With ≈35W of idle power draw, Zen 5 CPUs consume ≈3x as much power as Intel CPUs!

Intel’s CPUs offer great performance (like AMD), but excellent power consumption at idle.

Therefore, I can’t in good conscience buy an AMD CPU, but if you want a fast gaming-only PC or run an always-loaded HPC cluster with those CPUs, definitely go ahead :)

Graphics card: nvidia over AMD

I don’t necessarily recommend any particular nvidia graphics card, but I have had to stick to nvidia cards because they are the only option that work with my picky Dell UP3218K monitor.

From time to time, I try out different graphics cards. Recently, I got myself an AMD Radeon RX 9070 because I read that it works well with open source drivers.

While the Radeon RX 9070 works with my monitor (great!), it seems to consume 45W in idle, which is much higher than my nvidia cards, which idle at ≈ 20W. This is unacceptable to me: Aside from high power costs and wasting precious resources, the high power draw also means that my room will be hotter in summer and the fans need to spin faster and therefore louder.

People asked me on Social Media if this could be a measurement error (like, the card reporting inaccurate values), so I double-checked with a myStrom WiFi Switch and confirmed that with the Radeon card, the PC indeed draws 20-30W more from the wall socket.

Why Low Idle Power is so important

In the comments for my my previous blog post about the first build of this machine not running stable, people were asking why it is worth it to optimize a few watts of power usage. People calculate what higher power usage might cost, put it in relation to the total cost of the components, and conclude that saving ±10% of the price can’t possibly be worth the effort.

Let me try to illustrate the importance of low idle power with this anecdote: For one year, I was suffering from an nvidia driver bug that meant the GPU would not clock down to the most efficient power-saving mode (because of the high resolution of my monitor). The 10-20W of difference should have been insignificant. Yet, when the bug was fixed, I noticed how my PC got quieter (fans don’t need to spin up) and my room noticeably cooled down, which was great as it was peak temperatures in summer.

To me, having a whisper-quiet computing environment that does not heat up my room is a great, actual, real-life, measurable benefit. Not wasting resources and saving a tiny amount of money is a nice cherry on top.

Obviously all the factors are very dependent on your specific situation: Your house’s thermal behavior might differ from mine, your tolerance for noise (and/or baseline noise levels) might be different, you might put more/less weight on resource usage, etc.

Installation

UEFI setup

On the internet, I read that there was some issue related to the Power Limits that mainboards come with by default. Therefore, I did a UEFI firmware update immediately after getting the mainboard. I upgraded to version 1404 (2025/01/10) using the provided ZIP file (PRIME-Z890-P-ASUS-1404.zip) on an MS-DOS FAT-formatted USB stick with the EZ Flash tool in the UEFI firmware interface. Tip: do not extract the ZIP file, otherwise the EZ Flash tool cannot update the Intel ME firmware. Just put the ZIP file onto the USB disk as-is.

I verified that with this UEFI version, the Power Limit 1 (PL1) is 250W, and ICCMAX=347A, which are exactly the values that Intel recommends. Great!

I also enabled XMP and verified that memtest86 reported no errors.

Software setup: early adopter pains

To copy over the data from the old disk to the new disk, I wanted to boot a live linux distribution (specifically, grml.org) and follow my usual procedure: boot with the old disk and the new (empty) disk, then use dd to copy the data. It’s nice and simple, hard to screw up.

Unfortunately, while grml 2024.12 technically does boot up, there are two big problems:

There is no network connectivity because the kernel and linux-firmware versions are too old.
- Kernel commit r8169: add support for RTL8125D is not included.
I could not get Xorg to work at all. Not with the Intel integrated GPU, nor with the nvidia dedicated GPU. Not with nomodeset or any of the other options in the grml menu. This wasn’t merely a convenience problem: I needed to use gparted (the graphical version) for its partition moving/resizing support.

Ultimately, it was easier to upgrade my old PC to Linux 6.13 and linux-firmware 20250109, then put in the new disk and copy over the installation.

TRIM your SSDs

SSD disks can degrade over time, so it is essential that the Operating System tells the SSD firmware about freed-up blocks (for wear leveling). When using full-disk encryption, all involved layers need to have TRIM support enabled.

I think I saw the effect of an incorrectly configured TRIM setup in action back in 2022, when I copied my data from a Force MP600 to a WD Black SN850, which unexpectedly took many hours!

To make sure my disk has a long and healthy life, I double-checked that both periodic and continuous TRIM are enabled on my Arch Linux system: The fstab(5) file contains the discard option (and mount(8) lists the discard option), and fstrim.service ran within the last week:

systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
fstrim[779617]: /boot: 10.1 GiB (10799427584 bytes) trimmed on /dev/nvme0n1p1
fstrim[779617]: /: 1.8 TiB (2018906263552 bytes) trimmed on /dev/mapper/cryptroot
systemd[1]: fstrim.service: Deactivated successfully.

Speaking of copying data: the transfer from my WD Black SN850 to my Samsung 990 PRO ran at 856 MB/s and took about 40 minutes in total.

Performance

Here are the total times for a couple of typical workloads I run:

Workload	12900K (2022)	285K (2025)
build Go 1.24.3 (`cd src; ./make.bash`)	≈35s	≈26s
gokrazy/rsync tests (`make test`)	≈0.5s	≈0.4s
gokrazy UEFI test (`go test ./integration/...`)	≈30s	≈10s
gokrazy Linux compile (`gokr-rebuild-kernel -cross=arm64`)	3m 13s	2m 7s

The performance boost is great! Building Linux kernels a whole minute faster is really nice.

Stability issues

In March, I published an article about how the first build of this machine was not stable, in which you can read in detail about the various crashes I ran into.

Now, in early May, I know for sure that the CPU was defective, after a lengthy trouble-shooting in which I swapped out all the other parts of this PC, sent back the CPU and got a new one.

The CPU was the most annoying component to diagnose in this build because it’s an LGA 1851 socket and I don’t (yet) have any other machines which uses that same socket. AMD’s approach of sticking to each socket for a longer time would have been better in this situation.

Stress testing

When I published my earlier blog post about the PC being unstable, I did not really know how to reliably trigger the issue. Some compute-intensive tasks like running a Django test suite seemed to trigger the issue. I suspect that the problem somehow got worse, because when I started stress testing the machine, suddenly it would crash every time when building a Linux kernel.

That got me curious to see if other well-known CPU stress testers like Prime95 would show problems, and indeed: within seconds, Prime95 would report errors.

I figured I would use Prime95 as a quick signal: if it reports errors, the machine is faulty. This typically happens within seconds of starting Prime95.

If Prime95 reported no errors, I would use Linux kernel compilation as a slow signal: if I can successfully build a kernel, the machine is likely stable enough.

The specific setup I used is to run ./mprime -m, hit N (do not participate in distributed computation projects), then Enter a few times to confirm the defaults. Eventually, Prime95 starts calculating, which pushes the CPU to 100% usage (see the dstat(1) -like output by my gokrazy/stat implementation) and draws the expected ≈300W of power from the wall:

In addition, I also ran MemTest86 for a few hours:

To be clear: I also successfully ran MemTest86 on the previous, unstable build, so only running MemTest86 is not good enough if you are dealing with a faulty CPU.

RMA timeline

Jan 15th: I receive the components for my new PC
- In January and February, the PC crashes occasionally.
Mar 4th: I switch back to my old PC and start writing my blog post
Mar 19th: I publish my blog post about the machine not being stable
- The online discussion does not result in any interesting tips or leads.
Mar 20th: I order the AsRock Z890 Pro-A mainboard to ensure the mainboard is OK
Mar 24th: the AsRock Z890 Pro-A arrives
Apr 5th (Sat): started an RMA for the CPU
- They ask me to send the CPU to orderflow, which is the merchant that fulfilled my order.
- Typically, I prefer buying directly at digitec, but many PC components seem to only be available from orderflow on digitec nowadays.
Apr 9th (Wed): package arrives at orderflow (digitec gave me a non-priority return label)
Apr 14th (Mon): I got the following mail from digitec’s customer support and had to explain that I have thoroughly diagnosed the CPU as defective (a link to my blog post was sufficient):

Händler hat dies beim Hersteller angemeldet und dieser hat folgende Fragen:

Um sicherzugehen, dass wir Sie richtig verstehen: Sie haben die CPU auf zwei verschiedenen Motherboards getestet und das gleiche Problem besteht weiterhin?

Könnten Sie uns mitteilen, welche Marke und welches Modell die beiden verwendeten Motherboards sind?

Wurde auf beiden Motherboards die neueste BIOS-Version verwendet?

Bestand das Problem von Anfang an oder trat es erst später auf?

Wurde der Prozessor übertaktet? (Bitte beachten Sie, dass durch Übertakten die Garantie erlischt.)

Apr 25th (Fri): orderflow hands the replacement CPU to Swiss Post
May 1st (Thu): the machine successfully passes stress tests; I start using it

In summary, I spent March without a working PC, but that was because I didn’t have much time to pursue the project. Then, I spent April without a working PC because RMA’ing an Intel CPU through digitec seems pretty slow. I would have wished for a little more trust and a replacement CPU right away.

Conclusion

What a rollercoaster and time sink this build was! I have never received a faulty-on-arrival CPU in my entire life before. How did the CPU I first received pass Intel’s quality control? Or did it pass QC, but was damaged in transport? I will probably never know.

From now on, I know to extensively stress test new PC builds for stability to detect such issues quicker. Should the CPU be faulty, unfortunately getting it replaced is a month-long process — it’s very annoying to have such a costly machine just gather dust for a month.

But, once the faulty component was replaced, this is my best PC build yet:

The case is the perfect size for the components and offers incredibly convenient access to all components throughout the entire lifecycle of this PC, including the troubleshooting period, and the later stages of its life when this PC will be rotated into its “lab machine” period before I sell it second-hand to someone who will hopefully use the machine for another few years.

The machine is quiet, draws little power (for such a powerful machine) and really packs a punch!

As usual, I run Linux on this PC and haven’t noticed any problems in my day-to-day usage. I use suspend-to-RAM multiple times a day without any issues.

I hope some of these details were interesting and useful to you in your own PC builds!

If you want to learn about which peripherals I use aside from my 8K monitor (e.g. the Kinesis Advantage keyboard, Logitech MX Ergo trackball, etc.), check out my post stapelberg uses this: my 2020 desk setup. I might publish an updated version at some point :)

In praise of grobi for auto-configuring X11 monitors

2025-05-10T08:24:00+02:00

I have recently started using the grobi program by Alexander Neumann again and was delighted to discover that it makes using my fiddly (but wonderful) Dell 32-inch 8K monitor (UP3218K) monitor much more convenient — I get a signal more quickly than with my previous, sleep-based approach.

Previously, when my PC woke up from suspend-to-RAM, there were two scenarios:

The monitor was connected. My sleep program would power on the monitor (if needed), sleep a little while and then run xrandr(1) to (hopefully) configure the monitor correctly.
The monitor was not connected, for example because it was still connected to my work PC.

In scenario ②, or if the one-shot configuration attempt in scenario ① fails, I would need to SSH in from a different computer and run xrandr manually so that the monitor would show a signal:

% DISPLAY=:0 xrandr \
  --output DP-4 --mode 3840x4320 --panning 0x0+0+0 \
  --output DP-2 --right-of DP-4 --mode 3840x4320 --panning 0x0+3840+0

Automatic monitor configuration with grobi

I have now completely solved this problem by creating the following ~/.config/grobi.conf file:

rules:
  - name: UP3218K

    outputs_connected: [DP-2, DP-4]

	# DP-4 is left, DP-2 is right
    configure_row:
        - DP-4@3840x4320
        - DP-2@3840x4320

    # atomic instructs grobi to only call xrandr once and configure all the
    # outputs. This does not always work with all graphic cards, but is
	# needed to successfully configure the UP3218K monitor.
    atomic: true

…and installing / enabling grobi (on Arch Linux) using:

% sudo pacman -S grobi
% systemctl --user enable --now grobi

Whenever grobi detects that my monitor is connected (it listens for X11 RandR output change events), it will run xrandr(1) to configure the monitor resolution and positioning.

To check what grobi is seeing/doing, you can use:

% systemctl --user status grobi
% journalctl --user -u grob

For example, on my system, I see:

grobi: 18:31:48.823765 outputs: [HDMI-0 (primary) DP-0 DP-1 DP-2 (connected) 3840x2160+ [DEL-16711-808727372-DELL UP3218K-D2HP805I043L] DP-3 DP-4 (connected) 3840x21>
grobi: 18:31:48.823783 new rule found: UP3218K
grobi: 18:31:48.823785 enable outputs: [DP-4@3840x4320 DP-2@3840x4320]
grobi: 18:31:48.823789 using one atomic call to xrandr
grobi: 18:31:48.823806 running command /usr/bin/xrandr xrandr --output DP-4 --mode 3840x4320 --output DP-2 --mode 3840x4320 --right-of DP-4
grobi: 18:31:49.285944 new RANDR change event received

Notably, the instructions for getting out of a bad state (no signal) are now to power off the monitor and power it back on again. This will result in RandR output change events, which will trigger grobi, which will run xrandr, which configures the monitor. Nice!

Why not autorandr?

No particular reason. I knew grobi.

If nothing else, grobi is written in Go, so it’s likely to keep working smoothly over the years.

Does grobi work on Wayland?

Probably not. There is no mention of Wayland over on the grobi repository.

Bonus: my Suspend-to-RAM setup

As a bonus, this section describes the other half of my monitor-related automation.

When I suspend my PC to RAM, I either want to wake it up manually later, for example by pressing a key on the keyboard or by sending a Wake-on-LAN packet, or I want it to wake up automatically each morning at 6:50 — that way, daily cron jobs have some time to run before I start using the computer.

To accomplish this, I use zleep, a wrapper program around rtcwake(8) and systemctl suspend that integrates with the myStrom switch smart plug to turn off power to the monitor entirely. This is worthwhile because the monitor draws 30W even in standby!

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"os"
	"os/exec"
	"time"
)

var (
	resume = flag.Bool("resume",
		false,
		"run resume behavior only (turn on monitor via smart plug)")

	noMonitor = flag.Bool("no_monitor",
		false,
		"disable turning off/on monitor")
)

func monitorPower(ctx context.Context, method, cmnd string) error {
	if *noMonitor {
		log.Printf("[monitor power] skipping because -no_monitor flag is set")
		return nil
	}
	log.Printf("[monitor power] command: %v", cmnd)
	u, err := url.Parse("http://myStrom-Switch-A46FD0/" + cmnd)
	if err != nil {
		return err
	}
	for {
		if err := ctx.Err(); err != nil {
			return err
		}
		req, err := http.NewRequest(method, u.String(), nil)
		if err != nil {
			return err
		}
		ctx, canc := context.WithTimeout(ctx, 5*time.Second)
		defer canc()
		req = req.WithContext(ctx)
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			log.Print(err)
			time.Sleep(1 * time.Second)
			continue
		}
		if resp.StatusCode != http.StatusOK {
			log.Printf("unexpected HTTP status code: got %v, want %v", resp.Status, http.StatusOK)
			time.Sleep(1 * time.Second)
			continue
		}
		log.Printf("[monitor power] request succeeded")
		return nil
	}
}

func nextWakeup(now time.Time) time.Time {
	midnight := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.Local)
	if now.Hour() < 6 {
		// wake up today
		return midnight.Add(6*time.Hour + 50*time.Minute)
	}

	// wake up tomorrow
	return midnight.Add(24 * time.Hour).Add(6*time.Hour + 50*time.Minute)
}

func runResume() error {
	// Retry for up to one minute to give the network some time to come up
	ctx, canc := context.WithTimeout(context.Background(), 1*time.Minute)
	defer canc()
	if err := monitorPower(ctx, "GET", "relay?state=1"); err != nil {
		log.Print(err)
	}
	return nil
}

func zleep() error {
	ctx := context.Background()

	now := time.Now().Truncate(1 * time.Second)
	wakeup := nextWakeup(now)
	log.Printf("now   : %v", now)
	log.Printf("wakeup: %v", wakeup)
	log.Printf("wakeup: %v (timestamp)", wakeup.Unix())

	// assumes hwclock is running in UTC (see timedatectl | grep local)

	// Power the monitor off in 15 seconds.
	// mode=on is intentional: https://api.mystrom.ch/#e532f952-36ea-40fb-a180-a57b835f550e
	// - the switch will be turned on (already on, so this is a no-op)
	// - the switch will wait for 15 seconds
	// - the switch will be turned off
	if err := monitorPower(ctx, "POST", "timer?mode=on&time=15"); err != nil {
		log.Print(err)
	}

	sleep := exec.Command("sh", "-c", fmt.Sprintf("sudo rtcwake -m no --verbose --utc -t %v && sudo systemctl suspend", wakeup.Unix()))
	sleep.Stdout = os.Stdout
	sleep.Stderr = os.Stderr
	fmt.Printf("running %v\n", sleep.Args)
	if err := sleep.Run(); err != nil {
		return fmt.Errorf("%v: %v", sleep.Args, err)
	}

	return nil
}

func main() {
	flag.Parse()
	if *resume {
		if err := runResume(); err != nil {
			log.Fatal(err)
		}
	} else {
		if err := zsleep(); err != nil {
			log.Fatal(err)
		}
	}
}

To turn power to the monitor on after resuming, I placed the following shell script in /lib/systemd/system-sleep/zleep.sh:

#!/bin/sh

case "$1" in
	pre)	exit 0
		;;
	post)	/usr/local/bin/zleep -resume
		exit 0
		;;
 	*)	exit 1
		;;
esac

Once power is on, grobi will detect and configure the monitor.

Here is the program in action:

2025/05/06 21:58:32 now   : 2025-05-06 21:58:32 +0200 CEST
2025/05/06 21:58:32 wakeup: 2025-05-07 06:50:00 +0200 CEST
2025/05/06 21:58:32 wakeup: 1746593400 (timestamp)
2025/05/06 21:58:32 [monitor power] command: timer?mode=on&time=15
2025/05/06 21:58:32 [monitor power] request succeeded
running [sh -c sudo rtcwake -m no --verbose --utc -t 1746593400 && sudo systemctl suspend]
Using UTC time.
	delta   = 0
	tzone   = 0
	tzname  = UTC
	systime = 1746561512, (UTC) Tue May  6 19:58:32 2025
	rtctime = 1746561512, (UTC) Tue May  6 19:58:32 2025
alarm 1746593400, sys_time 1746561512, rtc_time 1746561512, seconds 0
rtcwake: wakeup using /dev/rtc0 at Wed May  7 04:50:00 2025
suspend mode: no; leaving

Intel 9 285K on ASUS Z890: not stable!

2025-03-19T17:35:38+01:00

In January I ordered the components for a new PC and expected that I would publish a successor to my 2022 high-end Linux PC 🐧 article. Instead, I am now sitting on a PC which regularly encounters crashes of the worst-to-debug kind, so I am publishing this article as a warning for others in case you wanted to buy the same hardware.

Components

Which components did I pick for this build? Here’s the full list:

Price	Type	Article
140 CHF	Case	Fractal Define 7 Compact Black Solid
155 CHF	Power Supply	Corsair RM850x
233 CHF	Mainboard	ASUS PRIME Z890-P
620 CHF	CPU	Intel Core Ultra 9 285k
120 CHF	CPU fan	Noctua NH-D15 G2
39 CHF	Case fan	Noctua NF-A14 PWM (140 mm)
209 CHF	RAM	64 GB DDR5-6400 Corsair Vengeance (2 x 32GB)
280 CHF	Disk	4000 GB Samsung 990 Pro
940 CHF	GPU	Inno3D GeForce RTX4070 Ti

Total: ≈1800 CHF, excluding the Graphics Card I re-used from a previous build.

…and the next couple of sections go into detail on how I selected these components.

Case

I really like building components into the case and working with the case. There are no sharp edges, the mechanisms are a pleasure to use and the cable-management is well thought-out.

Power Supply

I wanted to keep my options open regarding upgrading to an nVidia 50xx series graphics card at a later point. Those models have a TGP (“Total Graphics Power”) of 575 watts, so I needed a power supply that delivers enough power for the whole system even at peak power usage in all dimensions.

I ended up selecting the Corsair RM850x, which reviews favoribly (“leader in the 850W gold category”) and was available at my electronics store of choice.

This was a good choice: the PSU indeed runs quiet, and I really like the power cables (e.g. the GPU cable) that they include: they are very flexible, which makes them easy to cable-manage.

SSD disk

Instead, I picked the Samsung 990 Pro with 4 TB. I made good experiences with the Samsung Pro series over the years (never had one die or degrade performance), and my previous 2 TB disk is starting to fill up, so the extra storage space is appreciated.

Mainboard

In the 2.5 GbE space, the main players seem to be Realtek and Intel. Most mainboard vendors opted for Realtek as far as I could see.

Linux includes the r8169 driver for Realtek network cards, but you need a recent-enough Linux version (6.13+) that includes commit “r8169: add support for RTL8125D”, accompanied by a recent-enough linux-firmware package. Even then, there is some concern around stability and ASPM support. See for example this ServerFault post by someone working on the r8169 driver.

Despite the Intel 1 GbE options being well-supported at this point, Intel’s 2.5 GbE options might not fare any better than the Realtek ones: I found reports of instability with Intel’s 2.5 GbE network cards.

Aside from the network cards, I decided to stick to the ASUS prime series of mainboards, as I made good experiences with those in my past few builds. Here are a couple of thoughts on the ASUS PRIME Z890-P mainboard I went with:

I like the quick-release PCIe mechanism: ASUS understood that people had trouble unlocking large graphics cards from their PCIe slot, so they added a lever-like mechanism that is easily reachable. In my couple of usages, this worked pretty well!
I wrote about slow boot times with my 2022 PC build that were caused by time-consuming memory training. On this ASUS board, I noticed that they blink the Power LED to signal that memory training is in progress. Very nice! It hadn’t occurred to me previously that the various phases of the boot could be signaled by different Power LED blinking patterns :)
- The downside of this feature is: While the machine is in suspend-to-RAM, the Power LED also blinks! This is annoying, so I might just disconnect the Power LED entirely.
The UEFI firmware includes what they call a Q-Dashboard: An overview of what is installed/connected in which slot. Quite nice:

CPU fan

I am a long-time fan of Noctua’s products: This company makes silent fans with great cooling capacity that work reliably! For many years, I have swapped out every of my PC’s fans with Noctua fans, and it was always an upgrade. Highly recommended.

Hence, it is no question that I picked the latest and greatest Noctua CPU cooler for this build: the Noctua NH-D15 G2. There are a couple of things to pay attention to with this cooler:

I decided to configure it with one fan instead of two fans: Using only one fan will be the quietest setup, yet still have plenty of cooling capacity for this setup.
There are 3 different versions that differ in how their base plate is shaped. Noctua recommends: “For LGA1851, we generally recommend the regular standard version with medium base convexity” (https://noctua.at/en/intel-lga1851-all-you-need-to-know)
The height of this cooler is 168 mm. This fits well into the Fractal Define 7 Compact Black.

CPU

Probably the point that raises most questions about this build is why I selected an Intel CPU over an AMD CPU. The primary reason is that Intel CPUs are so much better at power saving!

Intel’s CPUs offer great performance (like AMD), but excellent power consumption at idle.

Therefore, I can’t in good conscience buy an AMD CPU, but if you want a fast gaming-only PC or run an always-loaded HPC cluster with those CPUs, definitely go ahead :)

Graphics card

I don’t necessarily recommend any particular nVidia graphics card, but I have had to stick to nVidia cards because they are the only option that work with my picky Dell UP3218K monitor.

From time to time, I try out different graphics cards. Recently, I got myself an AMD Radeon RX 9070 because I read that it works well with open source drivers.

While the Radeon RX 9070 works with my monitor (great!), it seems to consume 45W in idle, which is much higher than my nVidia cards, which idle at ≈ 20W. This is unacceptable to me: Aside from high power costs and wasting precious resources, the high power draw also means that my room will be hotter in summer and the fans need to spin faster and therefore louder.

Maybe I’ll write a separate article about the Radeon RX 9070.

Installation

UEFI setup

On the internet, I read that there was some issue related to the Power Limits that mainboards come with by default. Therefore, I did a UEFI firmware update first thing after getting the mainboard. I upgraded to version 1404 (2025/01/10) using the provided ZIP file (PRIME-Z890-P-ASUS-1404.zip) on an MS-DOS FAT-formatted USB stick with the EZ Flash tool in the UEFI firmware interface. Tip: do not extract the ZIP file, otherwise the EZ Flash tool cannot update the Intel ME firmware. Just put the ZIP file onto the USB disk as-is.

I verified that with this UEFI version, the Power Limit 1 (PL1) is 250W, and ICCMAX=347A, which are exactly the values that Intel recommends. Great!

I also enabled XMP and verified that memtest86 reported no errors.

Software setup: early adopter pains

Unfortunately, while grml 2024.12 technically does boot up, there are two big problems:

There is no network connectivity because the kernel and linux-firmware versions are too old.
- r8169: add support for RTL8125D
I could not get Xorg to work at all. Not with the Intel integrated GPU, nor with the nVidia dedicated GPU. Not with nomodeset or any of the other options in the grml menu. This wasn’t merely a convenience problem: I needed to use gparted (the graphical version) for its partition moving/resizing support.

Ultimately, it was easier to upgrade my old PC to Linux 6.13 and linux-firmware 20250109, then put in the new disk and copy over the installation.

Stability issues

At this point (early February), I switched to this new machine as my main PC.

Unfortunately, I could never get it to run stable! This journal shows you some of the issues I faced and what I tried to troubleshoot them.

Xorg dying after resume-from-suspend

One of the first issues I encountered with this system was that after resuming from suspend-to-RAM, I was greeted with a login window instead of my X11 session. The logs say:

(EE) NVIDIA(GPU-0): Failed to acquire modesetting permission.
(EE) Fatal server error:
(EE) EnterVT failed for screen 0
(EE) 
(EE) 
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE) 
(WW) NVIDIA(0): Failed to set the display configuration
(WW) NVIDIA(0):  - Setting a mode on head 0 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 1 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 2 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 3 failed: Insufficient permissions
(EE) Server terminated with error (1). Closing log file.

I couldn’t find any good tips online for this error message, so I figured I’d wait and see how frequently this happens before investigating further.

Feb 18: xHCI host controller dying

On Feb 18th, after resume-from-suspend, none of my USB peripherals would work anymore! This affected all USB ports of the machine and could not be fixed, not even by a reboot, until I fully killed power to the machine! In the kernel log, I saw the following messages:

xhci_hcd 0000:80:14.0: xHCI host not responding to stop endpoint command
xhci_hcd 0000:80:14.0: xHCI host controller not responding, assume dead
xhci_hcd 0000:80:14.0: HC died; cleaning up

Feb 24: xHCI host controller dying

The HC dying issue happened again when I was writing an SD card in my USB card reader:

xhci_hcd 0000:80:14.0: HC died; cleaning up

Feb 24: → UEFI update, disable XMPP

To try and fix the host controller dying issue, I updated the UEFI firmware to version 1601 and disabled the XMPP RAM profile.

Feb 26: → switch back from GeForce 4070 Ti to 3060 Ti

To rule out any GPU-specific issues, I decided to switch back from the Inno3D GeForce RTX4070 Ti to my older MSI GeForce RTX 3060 Ti.

Feb 28: PC dying on suspend-to-RAM

On Feb 28th, my PC did not resume from suspend-to-RAM. It would not even react to a ping, I had to hard-reset the machine. When checking the syslog afterwards, there are no entries.

I checked my power monitoring and saw that the machine consumed 50W (well above idle power, and far above suspend-to-RAM power) throughout the entire night. Hence, I suspect that the suspend-to-RAM did not work correctly and the machine never actually suspended.

Mar 4th: PC dying when running django tests

On March 4th, I was running the test suite for a medium-sized Django project (= 100% CPU usage) when I encountered a really hard crash: The machine stopped working entirely, meaning all peripherals like keyboard and mouse stopped responding, and the machine even did not respond to a network ping anymore.

At this point, I had enough and switched back to my 2022 PC.

Conclusion

What use is a computer that doesn’t work? My hierarchy of needs contains stability as the foundation, then speed and convenience. This machine exhausted my tolerance for frustration with its frequent crashes.

Manawyrm actually warned me about the ASUS board:

ASUS boards are a typical gamble as always – they fired their firmware engineers about 10 years ago, so you might get a nightmare of ACPI troubleshooting hell now (or it’ll just work). ASRock is worth a look as a replacement if that happens. Electronics are usually solid, though…

I didn’t expect that this PC would crash so hard, though. Like, if it couldn’t suspend/resume that would be one thing (a dealbreaker, but somewhat expected and understandable, probably fixable), but a machine that runs into a hard-lockup when compiling/testing software? No thanks.

I will buy a different mainboard to see if that helps, likely the ASRock Z890 Pro-A. If you have any recommendations for a Z890 mainboard that actually works reliably, please let me know!

Update 2025-04-17: I have received the ASRock Z890 Pro-A, but the machine shows exactly the same symptoms! I also swapped the power supply, which also did not help. Running Prime95 crashed almost immediately. At this point, I have to assume the CPU itself is defective and have started an RMA. I will post another update once (if?) I get a replaced CPU.

Update 2025-05-11: The CPU was faulty indeed! See My 2025 high-end Linux PC for a new article on this build, now with a working CPU.

Michael Stapelbergs Website

How my minimal, memory-safe Go rsync steers clear of vulnerabilities

Context: My own rsync

Security Vulnerabilities

January 2025 batch

CVE-2024-12084: Heap Buffer Overflow (9.8)

Stamp It! All Programs Must Report Their Version

Why are our versioning standards so low?!

Software Versioning

Case Study: i3’s --version and --moreversion

Coding Agent VMs on NixOS with microvm.nix

See also

Setting up microvm.nix

Step 1: network prep

Step 2: flake.nix

Step 3: microvm.nix

Step 4: microvm-base.nix

Can I finally start using Wayland in 2026?

Historical context

Making Wayland start

Hardware

nVidia driver support

Not working: TILE support for 8K monitor

Software: NixOS

Experiment results

Desktop: i3 → sway

Self-hosting my photos with Immich

Step 1. Hardware

Ryzen 7 Mini-PC makes a power-efficient VM host

Step 2. Install Immich

How I like to install NixOS (declaratively)

Step 2. Initial photos import

Step 3. Install the Immich iPhone App

Step 4. Backup

What’s missing?

Why Immich instead of…?

Conclusion

My impressions of the MacBook Pro M4

The nano-textured display

Specs: M4 or M4 Pro?

Impressions

120 Hz display

Conclusion

NixCon 2025 Trip Report 🐝

Friday

“What if GitHub Actions were local-first and built using Nix?”

“Rewriting the Hydra Queue Runner in Rust”

“You can’t spell “devshell” without “hell””

“The Nix Binary Cache and AWS”

“Nix-based development environments at Shopify (reprise)”

“My first Nix Aha!: A Newcomer’s Perspective”

Lightning Talks

Saturday

“The bikes have been shed: The official Nix formatter”

“Mastering NixOS Integration Tests: Advanced Techniques for Fast and Robust Multi-VM Tests”

“When Not to Nix: Working with External Config and SOPS Nix”

Lightning Talks

Sunday

Conclusion

Bye Intel, hi AMD! I’m done after 2 dead Intels

What happened? Or: the batch job of death

No, it wasn’t the heat wave

Which AMD CPU to buy?

Performance

Power consumption

Conclusion

Secret Management on NixOS with sops-nix

What is Secret Management?

sops-nix setup

Step 1. Preparation

Step 2. Obtain an age identity from your personal SSH key

Step 3. Obtain an age recipient for the remote machine

Step 4. Configure sops for your git repository

Step 5. Manage some secrets with sops

Step 6. Configure sops in NixOS

Usage Examples

Usage Example: command-line flags (ExecStart wrapper)

Usage Example: environment variable files

Usage Example: systemd credentials

Usage Example: samba users/passwords

Case Study: i3’s `--version` and `--moreversion`

Step 2: `flake.nix`

Step 3: `microvm.nix`

Step 4: `microvm-base.nix`