Michael Stapelbergs Website (Feed without AI posts)

Can I finally start using Wayland in 2026?

2026-01-04T08:55:00+01:00

Wayland is the successor to the X server (X11, Xorg) to implement the graphics stack on Linux. The Wayland project was actually started in 2008, a year before I created the i3 tiling window manager for X11 in 2009 — but for the last 18 years (!), Wayland was never usable on my computers. I don’t want to be stuck on deprecated software, so I try to start using Wayland each year, and this articles outlines what keeps me from migrating to Wayland in 2026.

Historical context

For the first few years, Wayland rarely even started on my machines. When I was lucky enough for something to show up, I could start some toy demo apps in the demo compositor Weston.

Around 2014, GNOME started supporting Wayland. KDE followed a few years later. Major applications (like Firefox, Chrome or Emacs) have been slower to adopt Wayland and needed users to opt into experimental implementations via custom flags or environment variables, until very recently, or — in some cases, like geeqie — still as of today.

Unfortunately, the driver support situation remained poor for many years. With nVidia graphics cards, which are the only cards that support my 8K monitor, Wayland would either not work at all or exhibit heavy graphics glitches and crashes.

In the 2020s, more and more distributions announced looking to switch to Wayland by default or even drop their X11 sessions, and RHEL is winding down their contributions to the X server.

Modern Linux distributions like Asahi Linux (for Macs, with their own GPU driver!) clearly consider Wayland their primary desktop stack, and only support X11 on a best-effort basis.

So the pressure to switch to Wayland is mounting! Is it ready now? What’s missing?

Making Wayland start

Hardware

I’m testing with my lab PC, which is a slightly upgraded version of my 2022 high-end Linux PC.

I describe my setup in more details in stapelberg uses this: my 2020 desk setup.

Most importantly for this article, I use a Dell 8K 32" monitor (resolution: 7680x4320!), which, in my experience, is only compatible with nVidia graphics cards (I try other cards sometimes).

Hence, both the lab PC and my main PC contain an nVidia GPU:

The lab PC contains a nVidia GeForce RTX 4070 Ti.
The main PC contains a nVidia GeForce RTX 3060 Ti.

(In case you’re wondering why I use the older card in my PC: I had a crash once where I suspected the GPU, so I switched back from the 4070 to my older 3060.)

nVidia driver support

For many years, nVidia drivers were entirely unsupported under Wayland.

Apparently, nVidia refused to support the API that Wayland was using, insisting that their EGLStreams approach was superior. Luckily, with nVidia driver 495 (late 2021), they added support for GBM (Generic Buffer Manager).

But, even with GBM support, while you could now start many Wayland sessions, the session wouldn’t run smoothly: You would see severe graphics glitches and artifacts, preventing you from getting any work done.

The solution for the glitches was explicit sync support: because the nVidia driver does not support implicit sync (like AMD or Intel), Wayland (and wlroots, and sway) needed to get explicit sync support.

Sway 1.11 (June 2025) and wlroots 0.19.0 are the first version with explicit sync support.

Not working: TILE support for 8K monitor

With the nVidia driver now working per se with Wayland, unfortunately that’s still not good enough to use Wayland in my setup: my Dell UP3218K monitor requires two DisplayPort 1.4 connections with MST (Multi Stream Transport) and TILE support. This combination worked just fine under X11 for the last 8+ years.

While GNOME successfully configures the monitor with its native resolution of 7680x4320@60, the monitor incorrectly shows up as two separate monitors in sway.

The reason behind this behavior is that wlroots does not support the TILE property (issue #1580 from 2019). Luckily, in 2023, contributor EBADBEEF sent draft merge request !4154, which adds support for the TILE property.

But, even with the TILE patch, my monitor would not work correctly: The right half of the monitor would just stay black. The full picture is visible when taking a screenshot with grim, so it seems like an output issue. I had a few exchanges about this with EBADBEEF starting in August 2025 (thanks for taking a look!), but we couldn’t figure out the issue.

A quarter later, I had made good experiences regarding debugging complex issues with the coding assistant Claude Code (Opus 4.5 at the time of writing), so I decided to give it another try. Over two days, I ran a number of tests to narrow down the issue, letting Claude analyze source code (of sway, wlroots, Xorg, mesa, …) and produce test programs that I could run manually.

Ultimately, I ended up with a minimal reproducer program (independent of Wayland) that shows how the SRC_X DRM property does not work on nVidia (but does work on Intel, for example!): I posted a bug report with a video in the nVidia forum and hope an nVidia engineer will take a look!

Crucially, with the bug now identified, I had Claude implement a workaround: copy the right half of the screen (at SRC_X=3840) to another buffer, and then display that buffer, but with SRC_X=0.

With that patch applied, for the first time, I can use Sway on my 8K monitor! 🥳

By the way, when I mentioned that GNOME successfully configures the native resolution, that doesn’t mean the monitor is usable with GNOME! While GNOME supports tiled displays, the updates of individual tiles are not synchronized, so you see heavy tearing in the middle of the screen, much worse than anything I have ever observed under X11. GNOME/mutter merge request !4822 should hopefully address this.

Software: NixOS

During 2025, I switched all my computers to NixOS. Its declarative approach is really nice for doing such tests, because you can reliably restore your system to an earlier version.

To make a Wayland/sway session available on my NixOS 25.11 installation, I added the following lines to my NixOS configuration file (configuration.nix):

# GDM display manager (can launch both X11/i3 and Wayland/Sway sessions)
services.displayManager.gdm.enable = true;
services.displayManager.gdm.autoSuspend = false;

# enable GNOME (for testing)
services.desktopManager.gnome.enable = true;

programs.sway = {
  enable = true;
  wrapperFeatures.gtk = true;
  extraOptions = [ "--unsupported-gpu" ];
};

I also added the following Wayland-specific programs to environment.systemPackages:

environment.systemPackages = with pkgs; [
  # …
  foot          # terminal emulator
  wtype         # replacement for xdotool type
  fuzzel        # fuzzy matching program starter
  wayland-utils # for wayland-info(1)
  gammastep     # redshift replacement
];

Note that activating this configuration kills your running X11 session, if any.

Just to be sure, I rebooted the entire machine after changing the configuration.

Experiment results

With this setup, I spent about one full work day in a Wayland session. Trying to actually get some work done uncovers issues that might not show in casual testing. Most of the day was spent trying to fix Wayland issues 😅. The following sections explain what I have learned/observed.

Desktop: i3 → sway

Many years ago, when Wayland became more popular, people asked on the i3 issue tracker if i3 would be ported to Wayland. I said no: How could I port a program to an environment that doesn’t even run on any of my computers? But also, I knew that with working a full-time job, I wouldn’t have time to be an early adopter and shape Wayland development.

This attitude resulted in Drew DeVault starting the Sway project around 2016, which aims to be a Wayland version of i3. I don’t see Sway as competition. Rather, I thought it was amazing that people liked the i3 project so much that they would go through the trouble of creating a similar program for other environments! What a nice compliment! 😊

Sway aims to be compatible with i3 configuration files, and it mostly is.

If you’re curious, here is what I changed from the Sway defaults, mostly moving key bindings around for the NEO keyboard layout I use, and configuring input/output blocks that I formerly configured in my ~/.xsession file:

my changes to the default Sway config

Self-hosting my photos with Immich

2025-11-29T08:22:05+01:00

For every cloud service I use, I want to have a local copy of my data for backup purposes and independence. Unfortunately, the gphotos-sync tool stopped working in March 2025 when Google restricted the OAuth scopes, so I needed an alternative for my existing Google Photos setup. In this post, I describe how I have set up Immich, a self-hostable photo manager.

Here is the end result: a few (live) photos from NixCon 2025:

Step 1. Hardware

I am running Immich on my Ryzen 7 Mini PC (ASRock DeskMini X600), which consumes less than 10 W of power in idle and has plenty of resources for VMs (64 GB RAM, 1 TB disk). You can read more about it in my blog post from July 2024:

Ryzen 7 Mini-PC makes a power-efficient VM host

When I saw the first reviews of the ASRock DeskMini X600 barebone, I was immediately interested in building a home-lab hypervisor (VM host) with it. Apparently, the DeskMini X600 uses less than 10W of power but supports latest-generation AMD CPUs like the Ryzen 7 8700G! Read more →

I installed Proxmox, an Open Source virtualization platform, to divide this mini server into VMs, but you could of course also install Immich directly on any server.

Step 2. Install Immich

I created a VM (named “photos”) with 500 GB of disk space, 4 CPU cores and 4 GB of RAM.

For the initial import, you could assign more CPU and RAM, but for normal usage, that’s enough.

I (declaratively) installed NixOS on that VM as described in this blog post:

How I like to install NixOS (declaratively)

For one of my network storage PC builds, I was looking for an alternative to Flatcar Container Linux and tried out NixOS again (after an almost 10 year break). There are many ways to install NixOS, and in this article I will outline how I like to install NixOS on physical hardware or virtual machines: over the network and fully declaratively. Read more →

Afterwards, I enabled Immich, with this exact configuration:

services.immich = {
  enable = true;
};

At this point, Immich is available on localhost, but not over the network, because NixOS enables a firewall by default. I could enable the services.immich.openFirewall option, but I actually want Immich to only be available via my Tailscale VPN, for which I don’t need to open firewall access — instead, I use tailscale serve to forward traffic to localhost:2283:

photos# tailscale serve --bg http://localhost:2283

Because I have Tailscale’s MagicDNS and TLS certificate provisioning enabled, that means I can now open https://photos.example.ts.net in my browser on my PC, laptop or phone.

Step 2. Initial photos import

At first, I tried importing my photos using the official Immich CLI:

% nix run nixpkgs#immich-cli -- login https://photos.example.ts.net secret
% nix run nixpkgs#immich-cli -- upload --recursive /home/michael/lib/photo/gphotos-takeout

Unfortunately, the upload was not running reliably and had to be restarted manually a few times after running into a timeout. Later I realized that this was because the Immich server runs background jobs like thumbnail creation, metadata extraction or face detection, and these background jobs slow down the upload to the extent that the upload can fail with a timeout.

The other issue was that even after the upload was done, I realized that Google Takeout archives for Google Photos contain metadata in separate JSON files next to the original image files:

Unfortunately, these files are not considered by immich-cli.

Luckily, there is a great third-party tool called immich-go, which solves both of these issues! It pauses background tasks before uploading and restarts them afterwards, which works much better, and it does its best to understand Google Takeout archives.

I ran immich-go as follows and it worked beautifully:

% immich-go \
  upload \
  from-google-photos \
  --server=https://photos.example.ts.net \
  --api-key=secret \
  ~/Downloads/takeout-*.zip

Step 3. Install the Immich iPhone App

My main source of new photos is my phone, so I installed the Immich app on my iPhone, logged into my Immich server via its Tailscale URL and enabled automatic backup of new photos via the icon at the top right.

I am not 100% sure whether these settings are correct, but it seems like camera photos generally go into Live Photos, and Recent should cover other files…?!

If anyone knows, please send an explanation (or a link!) and I will update the article.

I also strongly recommend to disable notifications for Immich, because otherwise you get notifications whenever it uploads images in the background. These notifications are not required for background upload to work, as an Immich developer confirmed on Reddit. Open Settings → Apps → Immich → Notifications and un-tick the permission checkbox:

Step 4. Backup

Immich’s documentation on backups contains some good recommendations. The Immich developers recommend backing up the entire contents of UPLOAD_LOCATION, which is /var/lib/immich on NixOS. The backups subdirectory contains SQL dumps, whereas the 3 directories upload, library and profile contain all user-uploaded data.

Hence, I have set up a systemd timer that runs rsync to copy /var/lib/immich onto my PC, which is enrolled in a 3-2-1 backup scheme.

What’s missing?

Immich (currently?) does not contain photo editing features, so to rotate or crop an image, I download the image and use GIMP.

To share images, I still upload them to Google Photos (depending on who I share them with).

Why Immich instead of…?

The two most promising options in the space of self-hosted image management tools seem to be Immich and Ente.

I got the impression that Immich is more popular in my bubble, and Ente made the impression on me that its scope is far larger than what I am looking for:

Ente is a service that provides a fully open source, end-to-end encrypted platform for you to store your data in the cloud without needing to trust the service provider. On top of this platform, we have built two apps so far: Ente Photos (an alternative to Apple and Google Photos) and Ente Auth (a 2FA alternative to the deprecated Authy).

I don’t need an end-to-end encrypted platform. I already have encryption on the transit layer (Tailscale) and disk layer (LUKS), no need for more complexity.

Conclusion

Immich is a delightful app! It’s very fast and generally seems to work well.

The initial import is smooth, but only if you use the right tool. Ideally, the official immich-cli could be improved. Or maybe immich-go could be made the official one.

I think the auto backup is too hard to configure on an iPhone, so that could also be improved.

But aside from these initial stumbling blocks, I have no complaints.

My impressions of the MacBook Pro M4

2025-10-31T11:04:59+01:00

I have been using a MacBook Pro M4 as my portable computer for the last half a year and wanted to share a few short impressions. As always, I am not a professional laptop reviewer, so in this article you won’t find benchmarks, just subjective thoughts!

Back in 2021, I wrote about the MacBook Air M1, which was the first computer I used that contained Apple’s own ARM-based CPU. Having a silent laptop with long battery life was a game-changer, so I wanted to keep those properties.

When the US government announced tariffs, I figured I would replace my 4-year old MacBook Air M1 with a more recent model that should last a few more years. Ultimately, Apple’s prices remained stable, so, in retrospect, I could have stayed with the M1 for a few more years. Oh well.

The nano-textured display

I went to the Apple Store to compare the different options in person. Specifically, I was curious about the display and whether the increased weight and form factor of the MacBook Pro (compared to a MacBook Air) would be acceptable. Another downside of the Pro model is that it comes with a fan, and I really like absolutely quiet computers. Online, I read from other MacBook Pro owners that the fan mostly stays off.

In general, I would have preferred to go with a MacBook Air because it has enough compute power for my needs and I like the case better (no ventilation slots), but unfortunately only the MacBook Pro line has the better displays.

Why aren’t all displays nano-textured? The employee at the Apple Store presented the trade-off as follows: The nano texture display is great at reducing reflections, at the expense of also making the picture slightly less vibrant.

I could immediately see the difference when placing two laptops side by side: The bright Apple Store lights showed up very prominently on the normal display (left), and were almost not visible at all on the nano texture display (right):

Personally, I did not perceive a big difference in “vibrancy”, so my choice was clear: I’ll pick the MacBook Pro over the MacBook Air (despite the weight) for the nano texture display!

After using the laptop in a number of situations, I am very happy with this choice. In normal scenarios, I notice no reflections at all (where my previous laptop did show reflections!). This includes using the laptop on a train (next to the window), or using the laptop outside in daylight.

Specs: M4 or M4 Pro?

(When I chose the new laptop, Apple’s M4 chips were current. By now, they have released the first devices with M5 chips.)

I decided to go with the MacBook Pro with M4 chip instead of the M4 Pro chip because I don’t need the extra compute, and the M4 needs less cooling — the M4 Pro apparently runs hotter. This increases the chance of the fan staying off.

Here are the specs I ended up with:

14" Liquid Retina XDR Display with nano texture
Apple M4 Chip (10 core CPU, 10 core GPU)
32 GB RAM (this is the maximum!), 2 TB SSD (enough for this computer)

Impressions

One thing I noticed is that the MacBook Pro M4 sometimes gets warm, even when it is connected to power, but is suspended to RAM (and has been fully charged for hours). I’m not sure why.

Luckily, the fan indeed stays silent. I think I might have heard it spin up once in half a year or so?

The battery life is amazing! The previous MacBook Air M1 had amazing all-day battery life already, and this MacBook Pro M4 lasts even longer. For example, watching videos on a train ride (with VLC) for 3 hours consumed only 10% of battery life. I generally never even carry the charger.

Because of that, Apple’s re-introduction of MagSafe, a magnetic power connector (so you don’t damage the laptop when you trip over it), is nice-to-have but doesn’t really make much of a difference anymore. In fact, it might be better to pack a USB-C cable when traveling, as that makes you more flexible in how you use the charger.

120 Hz display

I was curious whether the 120 Hz display would make a difference in practice. I mostly notice the increased refresh rate when there are animations, but not, for example, when scrolling.

One surprising discovery (but obvious in retrospect) is that even non-animations can become faster. For example, when running a Go web server on localhost, I noticed that navigating between pages by clicking links felt faster on the 120 Hz display!

The following illustration shows why that is, using a page load that takes 6ms of processing time. There are three cases (the illustration shows an average case and the worst case):

Best case: Page load finishes just before the next frame is displayed: no delay.
Worst case: Page load finishes just after a frame is displayed: one frame of delay.
Most page loads are somewhere in between. We’ll have 0.x to 1.0 frames of delay

As you can see, the waiting time becomes shorter when going from 60 Hz (one frame every 16.6ms) to 120 Hz (one frame every 8.3ms). So if you’re working with a system that has <8ms response times, you might observe actions completing (up to) twice as fast!

I don’t notice going back to 60 Hz displays on computers. However, on phones, where a lot more animations are a key part of the user experience, I think 120 Hz displays are more interesting.

Conclusion

My ideal MacBook would probably be a MacBook Air, but with the nano-texture display! :)

I still don’t like macOS and would prefer to run Linux on this laptop. But Asahi Linux still needs some work before it’s usable for me (I need external display output, and M4 support). This doesn’t bother me too much, though, as I don’t use this computer for serious work.

NixCon 2025 Trip Report 🐝

2025-09-21T09:34:00+02:00

I liked the NixOS meetup earlier this year, and at the end of the meetup they told everyone about NixCon 2025, which would be happening in Switzerland this year, at the very same location, the University Of Applied Sciences OST in Rapperswil, so I decided to go! In this trip report, I want to give you a rough impression of how I experienced this awesome conference :)

The bee in the title is a NixCon inside joke ;)

Friday

I arrived at about 09:30 on a rainy Friday morning, meaning I hurried from the train station into OST building 1 to show my ticket QR code and pick up my conference badge and custom name tag that I pre-ordered. The custom ones have your name engraved and come with a strong magnet to attach them to your clothes:

After grabbing a bite to eat, I headed to the main lecture hall for the opening session. Prof. Dr. Farhad Mehta from OST, as well as the entire NixCon orga team, welcomed the 450 registered attendees to the 10th NixCon! I recognized many familiar faces from the Nix meetup, but many hands went up when the audience was asked for whom it was the first time at NixCon, or in Switzerland in general.

I want to thank Prof. Mehta in particular for making possible such meetups and events! 👏

If you work at a university, school or other organisation that has access to rooms, consider offering to host a meetup (on a regular basis, or even just once)! Locations are always hard to find, so offering a space is a great contribution to Open Source.

“What if GitHub Actions were local-first and built using Nix?”

The first technical talk of the day was “What if GitHub Actions were local-first and built using Nix?” by Domen Kožar, the person behind cachix.org, which is a hosted Nix cache. The talk pitched cloud.devenv.sh, which is a Nix-based CI solution (like GitHub Actions) using devenv.

By using this solution, you solve the problem that you can’t easily / completely run GitHub Actions locally (yes, we all know about act), and you get to (?) write Nix configs instead of YAML configs.

The solution seems nice, but I found the talk a little unstructured because the presenter jumped around between slides so much. One crucial question was left unanswered: How do you integrate this custom solution with your GitHub projects? To me, diverging from the default way of configuring GitHub Actions does not seem worth it for my projects. YMMV.

→ watch the recording (46 minutes) on media.ccc.de

“Rewriting the Hydra Queue Runner in Rust”

Next up: “Rewriting the Hydra Queue Runner in Rust” by Simon Hauser from Helsinki Systems, a small German software company. Hydra is the component in the NixOS infrastructure which schedules builds: when nixpkgs changes, this is the component that runs the build whose result ends up on cache.nixos.org (the Debian equivalent is buildd).

Simon explained that bottlenecks in the current queue runner result in stranding of infrastructure: the project has machines available that it cannot use fully. He outlined how they replaced a crufty SSH-based automation with a well-designed gRPC protocol. I got the impression that a group of people was involved in developing and reviewing this design, which is a great sign for a healthy project.

One thing that was unfortunately missing from the talk were metrics. It would have been great to see a few graphs that illustrate just how much better the rewritten queue runner is.

Currently, the new queue runner is already used for Nix Community builds, but not yet in production for NixOS itself. Hopefully soon, though!

→ watch the recording (27 minutes) on media.ccc.de

“You can’t spell “devshell” without “hell””

This talk was presented by Zach Mitchell from Flox, which is a Nix-based dev environment solution. Thus far, I use nix-shell or nix develop (see Development shells with Nix: four quick examples), so I was curious what I’d learn from this talk.

Zach explained that both, nix-shell and nix develop were originally written to debug Nix package builds, not to provide general-purpose development environments. For users, this manifests as not being able to use your favorite shell — nix develop only supports Bash. One might read about nix develop -c exec , but that’s wrong, because then the shell’s RC files run after Nix setup, possibly destroying parts of the setup.

One interesting thing I learnt is that the Nix garbage collector scans /proc to avoid removing Nix store paths that are still needed by running processes.

Zach mentioned https://github.com/zmitchell/proctrace, which is a bpftrace-based profiler that tracks forks/execs and generates gantt chart syntax of the timing. Sounds cool, but is unfortunately broken right now…? Too bad.

→ watch the recording (45 minutes) on media.ccc.de

“The Nix Binary Cache and AWS”

In this fireside chat, Tarus Balog shared how he ended up at AWS after 20 years of Open Source, and how his team wants to give back to the community. One specific way in which they’re doing that is by hosting cache.nixos.org.

→ watch the recording (24 minutes) on media.ccc.de

“Nix-based development environments at Shopify (reprise)”

Josh Heinrichs from Shopify shared how they adopted Nix (again!), and I think real-world enterprise adoption stories like these are very interesting.

In summary, Shopify had a dev command (since 2016), which offered declarative configuration and then dispatched to apt (Linux) or homebrew (macOS). In the first attempt to move to Nix, the effort didn’t reach stable footing (some folks couldn’t use it yet) and then a company-wide shift to cloud development happened, where the easier solution was to “just use ubuntu”.

A few years in, folks are apparently not so happy with the cloud development environments and one day, Shopify CEO Tobias Lütke finds devenv, which is a Nix-based solution that is remarkably similar to Shopify’s dev. So Tobi adopts devenv for one of their services and becomes supportive of using Nix. This time around, they spend a lot more time on a successful rollout within the organization, meaning incremental adoption, getting all stakeholders on board, etc.

The takeaway is that one specific, well-supported use-case can be the adoption driver. And once you have your development environments on a Nix-based solution, you can more easily adopt other parts of the ecosystem as well.

→ watch the recording (19 minutes) on media.ccc.de

“My first Nix Aha!: A Newcomer’s Perspective”

In a similar spirit to the Shopify talk, Kavisha Kumar from ASML shared how she got into Nix after seeing a colleague use nix-shell to obtain a clean development shell.

Kavisha spent a lot of time at ASML to teach others about why and how to use Nix. She shared a number of nice metaphors that explained Nix concepts through the subject area of video gaming.

I think many people are excited about Nix, but have trouble conveying that excitement to others. Kavisha showed us a good way that worked for her.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day was filled with lightning talks.

Cole Mickens from Determinate Systems explained what features they are currently shipping in their downstream distribution “Determinate Nix” (features will be upstreamed): lazy trees (a performance optimization for evaluating Flakes), parallel evaluation (brings evaluation times down from 16s to 7s) and a native Linux builder for mac. Next up are Flake Schemas, which I haven’t read about yet.

Yvan Sraka from Numtide, a Nix and DevOps consultancy, showed how he manages Linux machines for friends and family with NixOS. He has his own configuration layer on top of NixOS and only uses the system as a base. Most actual programs are used through AppImage, Flatpaks, envfs and nix-ld. The latter two are solutions to use FHS based programs (those that expect /usr/bin and other standard locations to be present) on non-FHS systems like NixOS. I had heard of nix-ld before, but not of envfs.

Jacek Galowicz from Nixcademy showed how to use systemd-sysupdate and systemd-repart to implement A/B style updates with NixOS and systemd. It’s great to see that this technique is more and more mainstream, as I am also using A/B style updating successfully in gokrazy.

Saturday

The weather on Saturday was a lot better, so I made sure to get a seat with a view of Lake Zürich:

“The bikes have been shed: The official Nix formatter”

In this talk, Silvan Mosberger from Tweag (and one of the main NixCon organizers!), explains how the official formatting tool for .nix files came to be.

I was delighted to hear gofmt, the official Go formatter, being mentioned as a source of inspiration. Just like in other language ecosystems, introducing uniform formatting eliminates time-consuming back-and-forth in code review over adhering to coding style. Unfortunately, the formatting folks did not replicate one key aspect to gofmt’s success: gofmt has no options. As the famous Go proverb goes:

Gofmt’s style is no one’s favorite, yet gofmt is everyone’s favorite!

Meaning that it’s more important that everyone uses the same style, compared to everyone being able to express their personal style preferences.

→ watch the recording (20 minutes) on media.ccc.de

“Mastering NixOS Integration Tests: Advanced Techniques for Fast and Robust Multi-VM Tests”

In this two-hour workshop, Jacek Galowicz from Nixcademy, who is not only a Nix teacher, but also happens to be the maintainer of the NixOS integration test driver, shows us how to write complex integration tests with a few lines of Nix and Python.

Jacek showed an integration test example: a Bittorrent service, consisting of tracker, clients, firewalls and multiple networks! Nixpkgs contains over 1000 such integration tests, and running one on your laptop is easy.

The various ways to debug your tests seem pretty cool: using vsock instead of port forwardings, and enabling a debug hook that will make a failed test hang and wait to be debugged.

I thought this was a great overview and Jacek is an engaging teacher. I would recommend booking his classes!

“When Not to Nix: Working with External Config and SOPS Nix”

Ryota spoke about when to use Nix and when not to use Nix. For example, you could manage your dotfiles (config files) with Nix, or you could decide not to. Having recently migrated more and more machines and configurations to Nix, I found myself agreeing with this talk: It’s important to understand what you’ll get out of declaratively or statefully managed configs, and when which approach is better.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day I spent in lightning talks, some of which were sponsored talk slots. I learnt about, in no particular order:

Cloud Hypervisor, a KVM based hypervisor like qemu, but written in Rust.
nixbuild.net, a pay-as-you-go offering for extra build capacity you can rent. On Sunday I heard someone say that their company is using nixbuild.net and it’s very smooth.
NixCI, a Nix-based hosted CI. So, the cloud.devenv.sh service we heard about on Friday is a competitor to this service.
Nix in the Wild is an effort by Flox where they do 45-60 minute interviews about Nix success stories. This might help you convince folks in your organization.
clan is a fleet management solution.
NovaCustom, a one-person laptop/PC company. The laptops come with coreboot and work with NixOS.
ExpressVPN is migrating their internal server setup (TrustedServer) from Debian to NixOS! Deploying weekly in 105+ countries.
Cyberus, a German company, is offering NixOS LTS releases, compliant with the EU Cyber Resilience Act obligations.
David’s styx project is a more bandwidth-efficient download mechanism for NixOS updates. This uses EROFS, which seems like an interesting alternative to SquashFS images.

After all the talks, we met outside for a group picture followed by barbecue at the lake:

NixCon 2025 by Arik Grahl. Licensed under CC BY-SA 4.0.

Sunday

Before the conference, I wasn’t sure if I would even bother showing up for Sunday (Hack day), but on Sunday, I was like “of course!”, and it was a great decision!

Many people were still around and were working on their projects. It felt like the answer to any Nix question was just one chat message away — there was expertise and helping hands from many parts of the project.

I ended up meeting a couple of people I only knew from online interactions before, and we also talked a lot about meetups. Now, I am invited to multiple meetups to give a talk :D

Conclusion

This was a wonderful conference! The orga team and all contributors did a great job!

As always, the OST in Rapperswil is a great venue for Open Source events.

Ticket sales and talk submission / scheduling were done using the Pretix and Pretalx Open Source systems, which makes me proud to have contributed to Pretix.

The selection of talks was great: Some deeply technical, some covering only the human side of things, and many somewhere in between. I got the impression that all the presenters I saw genuinely cared about their topic, so the overall energy was very good!

(You can watch the talk recordings at media.ccc.de: NixCon 2025.)

Also outside of the talks, I had many friendly interactions and interesting conversations. There is a lot of interest and adoption of Nix, which is great to see!

The production level of the conference was very high for such a volunteer-driven event. For example, the very cool sounding break music between talks was created specifically for NixCon: “Lava” by tonstr.studio. Similarly, the welcome bag contained dark Swiss chocolate, specifically made for NixCon (see picture below). I don’t even like dark chocolate, but this one was delicious!

Thanks again to all helpers, and I look forward to coming back soon!

Bye Intel, hi AMD! I’m done after 2 dead Intels

2025-09-07T08:33:00+02:00

The Intel 285K CPU in my high-end 2025 Linux PC died again! 😡 Notably, this was the replacement CPU for the original 285K that died in March, and after reading through the reviews of Intel CPUs on my electronics store of choice, many of which (!) mention CPU replacements, I am getting the impression that Intel’s current CPUs just are not stable 😞. Therefore, I am giving up on Intel for the coming years and have bought an AMD Ryzen 9950X3D CPU instead.

What happened? Or: the batch job of death

On the 9th of July, I set out to experiment with layout-parser and tesseract in order to convert a collection of scanned paper documents from images into text.

I expected that offloading this task to the GPU would result in a drastic speed-up, so I attempted to build layout-parser with CUDA. Usually, it’s not required to compile software yourself on NixOS, but CUDA is non-free, so the default NixOS cache does not compile software with CUDA. (Tip: Enable the Nix Community Cache, which contains prebuilt CUDA packages, too!)

This lengthy compilation attempt failed with a weird symptom: I left for work, and after a while, my PC was no longer reachable over the network, but fans kept spinning at 100%! 😳 At first, I suspected a Linux bug, but now I am thinking this was the first sign of the CPU being unreliable.

When the CUDA build failed, I ran the batch job without GPU offloading instead. It took about 4 hours and consumed roughly 300W constantly. You can see it on this CPU usage graph (screenshot of a Grafana dashboard showing metrics collected by Prometheus):

On the evening of the 9th, the computer still seemed to work fine.

But the next day, when I wanted to wake up my PC from suspend-to-RAM as usual, it wouldn’t wake up. Worse, even after removing the power cord and waiting a few seconds, there was no reaction to pressing the power button.

Later, I diagnosed the problem to either the mainboard and/or the CPU. The Power Supply, RAM and disk all work with different hardware. I ended up returning both the CPU and the mainboard, as I couldn’t further diagnose which of the two is broken.

To be clear: I am not saying the batch job killed the CPU. The computer was acting strangely in the morning already. But the batch job might have been what really sealed the deal.

No, it wasn’t the heat wave

Tom’s Hardware recently reported that “Intel Raptor Lake crashes are increasing with rising temperatures in record European heat wave”, which prompted some folks to blame Europe’s general lack of Air Conditioning.

But in this case, I actually did air-condition the room about half-way through the job (at about 16:00), when I noticed the room was getting hot. Here’s the temperature graph:

I would say that 25 to 28 degrees celsius are normal temperatures for computers.

I also double-checked if the CPU temperature of about 100 degrees celsius is too high, but no: this Tom’s Hardware article shows even higher temperatures, and Intel specifies a maximum of 110 degrees. So, running at “only” 100 degrees for a few hours should be fine.

Lastly, even if Intel CPUs were prone to crashing under high heat, they should never die.

Which AMD CPU to buy?

I wanted the fastest AMD CPU (for desktops, not for servers), which currently is the Ryzen 9 9950X, but there is also the Ryzen 9 9950X3D, a variant with 3D V-Cache. Depending on the use-case, the variant with or without 3D V-Cache is faster, see the comparison on Phoronix.

Ultimately, I decided for the 9950X3D model, not just because it performs better in many of the benchmarks, but also because Linux 6.13 and newer let you control whether to prefer the CPU cores with larger V-Cache or higher frequency, which sounds like an interesting capability: By changing this setting, maybe one can see how sensitive certain workloads are to extra cache.

Aside from the CPU, I also needed a new mainboard (for AMD’s socket AM5), but I kept all the other components. I ended up selecting the ASUS TUF X870+ mainboard. I usually look for low power usage in a mainboard, so I made sure to go with an X870 mainboard instead of an X870E one, because the X870E has two chipsets (both of which consume power and need cooling)! Given the context of this hardware replacement, I also like the TUF line’s focus on endurance…

Performance

The performance of the AMD 9950X3D seems to be slightly better than the Intel 285K:

Workload	12900K (2022)	285K (2025)	9950X3D (2025)
build Go 1.24.3	≈35s	≈26s	≈24s
gokrazy/rsync tests	≈0.5s	≈0.4s	≈0.5s
gokrazy Linux compile	3m 13s	2m 7s	1m 56s

In case you’re curious, the commands used for each workload are:

cd src; ./make.bash
make test
gokr-rebuild-kernel -cross=arm64

(I have not included the gokrazy UEFI integration tests because I think there is an unrelated difference that prevents comparison of my old results with how the test runs currently.)

Power consumption

In my high-end 2025 Linux PC I explained that I chose the Intel 285K CPU for its lower idle power consumption, and some folks were skeptical if AMD CPUs are really worse in that regard.

Having switched between 3 different PCs, but with identical peripherals, I can now answer the question of how the top CPUs differ in power consumption!

I picked a few representative point-in-time power values from a couple of days of usage:

CPU	Mainboard	idle power	idle power with monitor
Intel 12900k	ASUS PRIME Z690-A	40W	60W
Intel 285k	ASUS PRIME Z890-P	46W	65W
AMD 9950X3D	ASUS TUF GAMING X870-PLUS WIFI	55W	80W

Looking at two typical evenings, here is the power consumption of the Intel 285K (measured using a myStrom WiFi switch smart plug, which comes with a REST API):

…and here is the same PC setup, but with the AMD 9950X3D:

I get the general impression that the AMD CPU has higher power consumption in all regards: the baseline is higher, the spikes are higher (peak consumption) and it spikes more often / for longer.

Looking at my energy meter statistics, I usually ended up at about 9.x kWh per day for a two-person household, cooking with induction.

After switching my PC from Intel to AMD, I end up at 10-11 kWh per day.

Conclusion

I started buying Intel CPUs because they allowed me to build high-performance computers that ran Linux flawlessly and produced little noise. This formula worked for me over many years:

Back in 2008, I bought a mobile Intel CPU in a desktop case (article in German).
Then, in 2012, I could just buy a regular Intel CPU (i7-2600K) for my Linux PC, because they had gotten so much better in terms of power saving.
Over the years, I bought an i7-8700K, and later an i9-9900K.
The last time this formula worked out for me was with my 2022 high-end Linux PC.

On the one hand, I’m a little sad that this era has ended. On the other hand, I have had a soft spot for AMD since I had one of their K6 CPUs in one of my early PCs and in fact, I have never stopped buying AMD CPUs (e.g. for my Ryzen 7-based Mini Server).

Maybe AMD could further improve their idle power usage in upcoming models? And, if Intel survives for long enough, maybe they succeed at stabilizing their CPU designs again? I certainly would love to see some competition in the CPU market.

Secret Management on NixOS with sops-nix

2025-08-24T09:56:00+02:00

Passwords and secrets like cryptographic key files are everywhere in computing. When configuring a Linux system, sooner or later you will need to put a password somewhere — for example, when I migrated my existing Linux Network Storage (NAS) setup to NixOS, I needed to specify the desired Samba passwords in my NixOS config (or manage them manually, outside of NixOS). For personal computers, this is fine, but if the goal is to share system configurations (for example in a Git repository), we need a different solution: Secret Management.

What is Secret Management?

The basic idea behind Secret Management systems is to encrypt the secrets at rest, meaning if somebody clones the git repository containing your NixOS system configurations, they cannot access (and therefore, also not deploy) the encrypted secrets.

Conceptually, we need to:

Encrypt the secrets such that the target system can decrypt them.
Encrypt the secrets such that other people working on this config can decrypt them.
Have the target system decrypt secrets at runtime.
Tell our software where to access the decrypted secrets.

sops-nix setup

In this article, I will show how to accomplish the above using sops-nix. Here’s a quick overview of the three different building blocks we will use:

sops is a tool to version-control secrets in git, in their encrypted form.
- sops makes it easy to re-encrypt these secrets when adding/removing authorized keys.
- sops is very flexible and can work with tons of other tools/providers.
sops-nix provides a way to integrate sops with Nix/NixOS
Using sops with age(1) allows us to use our existing SSH private key (humans) or SSH host private key (machines) instead of managing a separate set of key files.

You might wonder why I chose sops-nix over agenix, the other contender? The instructions for setting up sops-nix made more sense to me when I first looked at it, and I wanted to have the option to use sops in other ways, not just with age. If you’re curious about agenix, check out Andreas Gohr’s blog post about agenix.

Step 1. Preparation

I ran the following instructions on an Arch Linux machine on which I installed the Nix tool and enabled Nix Flakes. Follow the link for instructions also for other systems like Debian or Fedora.

Step 2. Obtain an age identity from your personal SSH key

I don’t want to manage an extra key file, so I’ll use ssh-to-age to derive a key from my SSH private key file, which I already take good care of to back up:

midna % mkdir -p $HOME/.config/sops/age/
midna % read -s SSH_TO_AGE_PASSPHRASE; export SSH_TO_AGE_PASSPHRASE
midna % nix run nixpkgs#ssh-to-age -- \
  -private-key \
  -i $HOME/.ssh/id_ed25519 \
  -o $HOME/.config/sops/age/keys.txt

(The SSH_TO_AGE_PASSPHRASE option is documented in the ssh-to-age README.)

To display the age recipient (public key) of this age identity (private key), I used:

midna % nix shell nixpkgs#age
midna 2 % age-keygen -y $HOME/.config/sops/age/keys.txt
age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf

Step 3. Obtain an age recipient for the remote machine

Similarly, I will derive an age recipient from the SSH host key of the remote system:

batchn % cat /etc/ssh/ssh_host_ed25519_key.pub | nix run nixpkgs#ssh-to-age
age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k

Step 4. Configure sops for your git repository

In my git repository (nix-configs), I have one subdirectory per NixOS system, i.e. tree(1) shows:

├── batchn
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
│   ├── secrets
│   │   └── example.yaml
├── wiki
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
…

In the root of the git repository (next to the batchn directory), I create .sops.yaml like so:

keys:
  - &admin_michael age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf
  - &server_batchn age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k
# …more server keys go here…
creation_rules:
  - path_regex: batchn/secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
    - age:
      - *admin_michael
      - *server_batchn

The more systems I manage, the more keys and creation_rules I will need to configure.

The creation rules tell sops which keys to use when encrypting a file. In my setups, I typically use only a single file per system, but I could imagine splitting out some secrets into a separate file if I wanted to collaborate with someone on just one aspect of the system.

Step 5. Manage some secrets with sops

Now that we told sops which recipients to encrypt for, we can decrypt and edit secrets/example.yaml in our configured editor by running:

midna ~/nix-configs/batchn % nix run nixpkgs#sops secrets/example.yaml

The simplest key file contains just one key, for example:

api-key: hello world :)

After saving and exiting your editor, sops will update the encrypted secrets/example.yaml.

Step 6. Configure sops in NixOS

Now, we need to reference the encrypted file in NixOS and enable sops-nix integration to make the decrypted secrets available on the system.

In flake.nix, I added sops-nix to the inputs section and added the NixOS module. I show the entire diff because the places where the lines go are just as important as what the lines say:

--- c/batchn/flake.nix
+++ i/batchn/flake.nix
@@ -1,85 +1,93 @@
 {
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

     disko.url = "github:nix-community/disko";
     # Use the same version as nixpkgs
     disko.inputs.nixpkgs.follows = "nixpkgs";

     stapelbergnix.url = "github:stapelberg/nix";

     zkjnastools.url = "github:stapelberg/zkj-nas-tools";

+    sops-nix = {
+      url = "github:Mic92/sops-nix";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
   };

   outputs =
     {
       nixpkgs,
       disko,
       stapelbergnix,
       zkjnastools,
+      sops-nix,
       ...
     }:
     let
       system = "x86_64-linux";
       pkgs = import nixpkgs {
         inherit system;
         config.allowUnfree = false;
       };
     in
     {
       nixosConfigurations.batchn = nixpkgs.lib.nixosSystem {
         inherit system;
         inherit pkgs;
         modules = [
           disko.nixosModules.disko
           ./configuration.nix
           stapelbergnix.lib.userSettings
           # Use systemd for network configuration
           stapelbergnix.lib.systemdNetwork
           # Use systemd-boot as bootloader
           stapelbergnix.lib.systemdBoot
           # Run prometheus node exporter in tailnet
           stapelbergnix.lib.prometheusNode
           zkjnastools.nixosModules.zkjbackup
+          sops-nix.nixosModules.sops
         ];
       };
       formatter.${system} = pkgs.nixfmt-tree;
     };
 }

Then, in configuration.nix, we tell sops-nix to use the SSH host key as identity, where sops will find our secrets and which secrets sops-nix should realize on the remote system:

  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  sops.defaultSopsFile = ./secrets/example.yaml;
  sops.secrets."api-key" = { };

After deploying, we can access the secret on the running system:

batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Of course, even after rebooting the machine, the secrets remain available without a re-deploy:

batchn ~ % uptime
 22:09:23  up   0:00,  1 user,  load average: 0,32, 0,08, 0,03
batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Usage Examples

Now that we have secrets stored in files under /run/secrets, how can we use these secrets?

The following sections show a few common ways.

Usage Example: command-line flags (ExecStart wrapper)

Let’s assume you have deployed a custom Go server as a systemd service on NixOS as follows, and you want to start managing the cleartext secret passed via the -securecookie_hash_key and -securecookie_block_key command-line flags:

{
  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="some-secret-key" \
          -securecookie_block_key="a-different-secret-key"
      '';
    };
  };
}

With the following sops secrets:

fortuneserver:
    securecookie_hash_key: some-secret-key
    securecookie_block_key: a-different-secret-key

…we need to adjust our NixOS config to read these secret files at runtime. Because the ExecStart directive is interpreted by systemd and not passed through a shell, we use the writeShellScript helper and then just cat the files:

{
  sops.secrets."fortuneserver/securecookie_hash_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };
  sops.secrets."fortuneserver/securecookie_block_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };

  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = pkgs.writeShellScript "fortuneserver-execstart" ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="$(cat /run/secrets/fortuneserver/securecookie_hash_key)" \
          -securecookie_block_key="$(cat /run/secrets/fortuneserver/securecookie_block_key)"
      '';
    };
  };
}

Usage Example: environment variable files

What if the service in question does not use command-line flags, but environment variables for configuring secrets? We can put an environment variable file into a sops-managed secret:

translate-fe:
    env: |
        DEEPL_AUTH_KEY=my-deepl-key

…and then we make systemd apply these environment variables from the secrets file:

{
  sops.secrets."translate-fe/env" = {
    owner = "translatefe";
    restartUnits = [ "translate-fe.service" ];
  };

  systemd.services.translate-fe = {
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "translatefe";

      EnvironmentFile = [ config.sops.secrets."translate-fe/env".path ];

      ExecStart = "${translatefeExecstart}/bin/translate-fe";
    };
  };
}

If you are configuring a NixOS module (instead of declaring a custom service), the option might not always be called EnvironmentFile. For example, for the oauth2-proxy service, you would need to configure the services.oauth2-proxy.keyFile option:

  services.oauth2-proxy = {
    keyFile = config.sops.secrets."oauth2-proxy/env".path;
    enable = true;
    # …
  };

Usage Example: systemd credentials

In the previous examples, we configured the owner of each secret to the user account under which the service is running. But what if there is no such user account, because the service use systemd’s DynamicUser feature?

We can use systemd’s LoadCredential feature! For example, I supply the SMTP password to my Prometheus Alertmanager as follows:

{
  sops.secrets."alertmanager/smtp_pw" = {
    restartUnits = [ "alertmanager.service" ];
  };

  systemd.services.alertmanager.serviceConfig.LoadCredential = [
    "smtp_pw:${config.sops.secrets."alertmanager/smtp_pw".path}"
  ];

  services.prometheus.alertmanager = {
    enable = true;

    configuration = {
      global = {
        smtp_smarthost = "smtp.gmail.com:587";
        smtp_from = "alerts@example.net";
        smtp_auth_username = "alerts@example.net";
        smtp_auth_password_file = "/run/credentials/alertmanager.service/smtp_pw";
      };

      # …remaining config goes here…
    };
  };
}

Usage Example: samba users/passwords

In my blog post “Migrating my NAS from CoreOS/Flatcar Linux to NixOS”, I describe how to configure samba users and passwords (from sops-managed secrets) with an ExecStartPre shell script (which is very similar to the techniques already explained).

Conclusion

Managing secrets as separately-encrypted files in your config repository makes sense to me!

age’s ability to work with SSH keys makes for a really convenient setup, in my opinion. Encrypting secrets for the destination system’s SSH host key feels very elegant.

I hope the examples above are sufficient for you to efficiently configure secrets in NixOS!

Development shells with Nix: four quick examples

2025-07-27T08:50:00+02:00

I wanted to use GoCV for one of my projects (to find and extract paper documents from within a larger scan), without permanently having OpenCV on my system.

This seemed like a good example use-case to demonstrate a couple of Nix commands I like to use, covering quick interactive one-off dev shells to fully declarative, hermetic, reproducible, shareable dev shells.

Notably, you don’t need to use NixOS to run these commands! You can install and use Nix on any Linux system like Debian, Arch, etc., as long as you set a Nix path or use Flakes (see setup).

For comparison: The Debian Way

Before we start looking at Nix, I will show how to get GoCV running on Debian.

Let’s create a minimal Go program which uses a GoCV function like gocv.NewMat(), just to verify that we can compile this program:

package main

import "gocv.io/x/gocv"

func main() {
  gocv.NewMat()
}

If we try to build this on a Debian system, we get:

debian % mkdir -p /tmp/minimal
debian % cd /tmp/minimal

debian % cat > minimal.go <<'EOT'
package main
import "gocv.io/x/gocv"
func main() { gocv.NewMat(); }
EOT

debian % go mod init minimal
go: creating new go.mod: module minimal
go: to add module requirements and sums:
	go mod tidy

debian % go mod tidy
go: finding module for package gocv.io/x/gocv
go: downloading gocv.io/x/gocv v0.41.0
go: found gocv.io/x/gocv in gocv.io/x/gocv v0.41.0

debian % go build
# gocv.io/x/gocv
# [pkg-config --cflags  -- opencv4]
Package opencv4 was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv4.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv4', required by 'virtual:world', not found

On Debian, we can install OpenCV as follows:

debian % sudo apt install libopencv-dev

[…]

Summary:
  Upgrading: 7, Installing: 512, Removing: 0, Not Upgrading: 27
  Download size: 367 MB
  Space needed: 1590 MB / 281 GB available

Continue? [Y/n]

Saying “yes” to this prompt downloads and installs over 500 packages (takes a few minutes).

Now the build works:

debian % go build
debian % file minimal
minimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), […]

…but we have over 500 extra packages on our system that will now need to be updated in all eternity, therefore I would like to separate this one-off experiment from my usual system.

We could use Docker to start a Debian container and work inside that container, but, depending on the task, this can be cumbersome precisely because it’s a separate environment. For this example, I would need to specify a volume mount to make my input files available to the Docker container, and I would need to set up environment variables before programs inside the Docker container can open graphical windows on the host…

Let’s look at how we can use Nix to help us with that!

Setup: Nix-on-Debian (or Nix-on-Arch, or…)

Users of NixOS can skip this section, as NixOS systems include a ready-to-use Nix.

Before you can try the examples on your own computer, you need to complete these three steps:

Install Nix
Enable Flakes
Set a Nix path

Step 1: Install Nix

Users of Debian, Arch, Fedora, or other Linux systems first need to install Nix. Luckily, Nix is available for many popular Linux distributions:

Debian ships nix-setup-systemd
Arch Linux packages nix and provides documentation on the Nix Arch Wiki page. In practice, I installed the package and configured a couple of nixbld users.
More generally, there are Nix builds (rpm, deb, pacman) available for a number of distributions: https://github.com/nix-community/nix-installers

Step 2: Enable Flakes

Nix flakes are “a generic way to package Nix artifacts”.

Examples 3 and 4 use Nix flakes to pin dependencies, so we need to enable Nix flakes.

Step 3: Set a Nix path

For example 1 and 2, we want to use the Nix expression import .

On NixOS, this expression will follow the system version, meaning if you use import on a NixOS 25.05 installation, that will reference nixpkgs in version nixos-25.05.

On other Linux systems, you’ll see an error message like this:

debian-server % nix-shell -p pkg-config opencv
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

       at «string»:1:25:

            1| {...}@args: with import  args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (pkg-config) (opencv) ]; } ""
             |                         ^
(use '--show-trace' to show detailed location information)

We need to tell Nix which version of nixpkgs to use by setting the Nix search path:

debian-server % export NIX_PATH=nixpkgs=channel:nixos-25.05
debian-server % nix-shell -p pkg-config opencv
[nix-shell:/tmp/opencv]#

Alright! Now we are set up. Let’s jump into the first example!

Example 1: Interactive one-offs: nix-shell

Nix provides a middle-ground between installing OpenCV on your system (apt install like in the example above) and installing OpenCV in a separate Docker container: Nix can make available OpenCV without permanently installing it.

We can run nix-shell(1) to start a bash shell in which the specified packages are available. To successfully build Go code that uses GoCV, we need to have OpenCV available:

% nix-shell -p pkg-config opencv
these 194 paths will be fetched (175.80 MiB download, 764.10 MiB unpacked):
  /nix/store/ig2nk0hsha9xaailhaj69yv677nv95q4-abseil-cpp-20210324.2
  /nix/store/yw5xqn8lqinrifm9ij80nrmf0i6fdcbx-alsa-lib-1.2.13
[…]

[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

In case you were wondering: Yes, we do need to specify pkg-config in this nix-shell command explicitly, otherwise running pkg-config will run the host version (outside the dev shell), which cannot find opencv4.pc.

Example 2: nix-shell config file: shell.nix

Once we have a combination of packages that work for our project (in our example, just pkg-config and opencv), we can create a shell.nix (in any directory, but usually in the root of a project) which nix-shell (without the -p flag) will read:

{
  pkgs ? import  { },
}:
pkgs.mkShell {
  packages = with pkgs; [
    # Explicitly list pkg-config so that mkShell will arrange
    # for the PKG_CONFIG_PATH to find the .pc files.
    pkg-config
    opencv
  ];
}

…and then, we just run nix-shell:

% nix-shell
[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

If you’re curious, here are a couple of documentation pointers regarding the boilerplate around the list of packages:

Line 1 to 3 declare a function with an argument set — this is the required structure for nix-shell to be able to call your shell.nix file.
pkgs.mkShell is a convenience helper for use with nix-shell.
The with pkgs; part allows us to write opencv instead of pkgs.opencv.

By the way: With the nixd language server, editors with LSP support can show the versions that packages resolve to, point out your spelling mistakes, or provide features like “jump to definition”.

For example, in this screenshot, I was editing shell.nix in Emacs and was curious how the Nix source of the opencv package looked like. By pressing M-. (xref-find-definitions) with “point” over opencv, I got to opencv/4.x.nix in my local Nix store:

Example 3: Hermetic, pinned devShells: Nix Flakes

The previous examples used nixpkgs from your system (or Nix path), which means you don’t need to change the .nix file when you upgrade your system — depending on the use-case, I see this behavior as either convenient or terrifying.

For use-cases where it is important that the .nix file is built exactly the same way, no matter what version the surrounding OS uses, we can use Nix Flakes to build in a hermetic way, with dependency versions pinned in the flake.lock file.

A flake.nix contains the same mkShell expression as above, but declares structure around it: The mkShell expression goes into the outputs.devShells.x86_64-linux.default attribute and the inputs attribute contains Flake references that are available to this build:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";

  outputs =
    { self, nixpkgs }:
    {
      devShells.x86_64-linux.default =
        let
          pkgs = nixpkgs.legacyPackages.x86_64-linux;
        in
        pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
    };
}

By the way: Despite the name, it is a best practice to use nixpkgs.legacyPackages, which conceptually provides a single import nixpkgs result (for efficiency).

Now, I can use nix develop to get a shell with OpenCV:

% nix develop
michael@midna$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

The first nix develop run creates a flake.lock file, so running nix develop later will get us exactly the same environment. To update to newer versions, use nix flake update.

Tip: Instead of a shell, nix develop --command=emacs is also a useful variant.

Example 4: Making the Flake system-independent

Unfortunately, the above flake.nix hard-codes x86_64-linux, so it will not be usable on, say, an aarch64-linux (ARM) computer, or on a x86_64-darwin (Mac).

Having to explicitly specify the system by default is a long-standing criticism of Nix Flakes.

There are a number of workarounds. For example, we can use numtide/flake-utils and refactor our flake.nix to use its eachDefaultSystem convenience function:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        formatter = pkgs.nixfmt-tree;
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
      }
    );
}

Or we could use numtide/blueprint, its spiritual successor.

LucPerkins’s dev-templates have effectively inlined a version of this technique.

For a solution that isn’t part of Nix, but Nix-adjacent: devenv is a separate tool that is built on Nix (no longer using the CppNix implementation, but tvix actually), but with its own .nix files.

Tip: Keeping packages around

If you notice that nix develop or similar commands fetch packages despite the flake.lock not having changed, you can install the Flake into your profile to declare it as a gcroot to Nix:

% nix profile install .#devShells.x86_64-linux.default

But wait, isn’t that getting us into the same state as with The Debian Way? No! While OpenCV will remain available indefinitely if you install the flake into your profile, there still is a layer of separation: Within your system, OpenCV isn’t available, only when you start a development shell with nix-shell or nix develop.

Conclusion

How do the four examples above compare? Here’s an overview:

Example	Boilerplate	Pinned?	System-dependent?
Ex 1: `nix-shell -p …`	😊	no	no
Ex 2: `shell.nix`	🙂	no	no
Ex 3: `flake.nix`	😲	yes	yes
Ex 4: system-independent `flake.nix`	🤨	yes	no

For personal one-off experiments, I use nix-shell.

Once the experiment works, I typically want to pin the dependencies, so I use a flake.nix.

If this is software that isn’t just versioned, but also published (or worked on with multiple people/systems), I go through the effort of making it a system-independent flake.nix.

I hope in the future, it will become easier to write a system-independent flake.

Despite the rough edges, I appreciate the reproducibility and control that Nix gives me!

Migrating my NAS from CoreOS/Flatcar Linux to NixOS

2025-07-13T08:17:00+02:00

In this article, I want to show how to migrate an existing Linux server to NixOS — in my case the CoreOS/Flatcar Linux installation on my Network Attached Storage (NAS) PC.

I will show in detail how the previous CoreOS setup looked like (lots of systemd units starting Docker containers), how I migrated it into an intermediate state (using Docker on NixOS) just to get things going, and finally how I migrated all units from Docker to native NixOS modules step-by-step.

If you haven’t heard of NixOS, I recommend you read the first page of the NixOS website to understand what NixOS is and what sort of things it makes possible.

The target audience of this blog post is people interested in trying out NixOS for the use-case of a NAS, who like seeing examples to understand how to configure a system.

You can apply these examples by first following my blog post “How I like to install NixOS (declaratively)”, then making your way through the sections that interest you. If you prefer seeing the full configuration, skip to the conclusion.

Context/History

Over the last decade, I used a number of different operating systems for my NAS needs. Here’s an overview of the 2 NAS systems storage2 and storage3:

Year	storage2	storage3	Details (blog post)
2013	Debian on qnap	Debian on qnap	Wake-On-LAN with Debian on a qnap TS-119P2+
2016	CoreOS on PC	CoreOS on PC	Gigabit NAS (running CoreOS)
2023	CoreOS on PC	Ubuntu+ZFS on PC	My all-flash ZFS NAS build
2025	NixOS on PC	Ubuntu+ZFS on PC	→ you are here ←
?	NixOS on PC	NixOS+ZFS on PC	Converting more PCs to NixOS seems inevitable ;)

My NAS Software Requirements

(This post is only about software! For my usage patterns and requirements regarding hardware selection, see “Design Goals” in my My all-flash ZFS NAS build post (2023).)
Remote management: I really like the model of having the configuration of my network storage builds version-controlled and managed on my main PC. It’s a nice property that I can regain access to my backup setup by re-installing my NAS from my PC within minutes.
Automated updates, with easy rollback: Updating all my installations manually is not my idea of a good time. Hence, automated updates are a must — but when the update breaks, a quick and easy path to recovery is also a must.
- CoreOS/Flatcar achieved that with an A/B updating scheme (update failed? boot the old partition), whereas NixOS achieves that with its concept of a “generation” (update failed? select the old generation), which is finer-grained.

Why migrate from CoreOS/Flatcar to NixOS?

When I started using CoreOS, Docker was pretty new technology. I liked that using Docker containers allowed you to treat services uniformly — ultimately, they all expose a port of some sort (speaking HTTP, or Postgres, or…), so you got the flexibility to run much more recent versions of software on a stable OS, or older versions in case an update broke something.

Over a decade later, Docker is established tech. People nowadays take for granted the various benefits of the container approach.

So, here’s my list of reasons why I wasn’t satisfied with Flatcar Linux anymore.

R1. cloud-init is deprecated

The CoreOS cloud-init project was deprecated at some point in favor of Ignition, which is clearly more powerful, but also more cumbersome to get started with as a hobbyist. As far as I can tell, I must host my config at some URL that I then provide via a kernel parameter. The old way of just copying a file seems to no longer be supported.

Ignition also seems less convenient in other ways: YAML is no longer supported, only JSON, which I don’t enjoy writing by hand. Also, the format seems to change quite a bit.

As a result, I never made the jump from cloud-init to Ignition, and it’s not good to be reliant on a long-deprecated way to use your OS of choice.

R2. Container Bitrot

At some point, I did an audit of all my containers on the Docker Hub and noticed that most of them were quite outdated. For a while, Docker Hub offered automated builds based on a Dockerfile obtained from GitHub. However, automated builds now require a subscription, and I will not accept a subscription just to use my own computers.

R3. Dependency on a central service

If Docker at some point ceases operation of the Docker Hub, I am unable to deploy software to my NAS. This isn’t a very hypothetical concern: In 2023, Docker Hub announced the end of organizations on the Free tier and then backpedaled after community backlash.

Who knows how long they can still provide free services to hobbyists like myself.

R4. Could not try Immich on Flatcar

The final nail in the coffin was when I noticed that I could not try Immich on my NAS system! Modern web applications like Immich need multiple Docker containers (for Postgres, Redis, etc.) and hence only offer Docker Compose as a supported way of installation.

Unfortunately, Flatcar does not include Docker Compose.

I was not in the mood to re-package Immich for non-Docker-Compose systems on an ongoing basis, so I decided that a system on which I can neither run software like Immich directly, nor even run Docker Compose, is not sufficient for my needs anymore.

Reason Summary

With all of the above reasons, I would have had to set up automated container builds, run my own central registry and would still be unable to run well-known Open Source software like Immich.

Instead, I decided to try NixOS again (after a 10 year break) because it seems like the most popular declarative solution nowadays, with a large community and large selection of packages.

How does NixOS compare for my situation?

Same: I also need to set up an automated job to update my NixOS systems.
- I already have such a job for updating my gokrazy devices.
- Docker push is asynchronous: After a successful push, I still need extra automation for pulling the updated containers on the target host and restarting the affected services, whereas NixOS includes all of that.
Better: There is no central registry. With NixOS, I can push the build result directly to the target host via SSH.
Better: The corpus of available software in NixOS is much larger (including Immich, for example) and the NixOS modules generally seem to be expressed at a higher level of abstraction than individual Docker containers, meaning you can configure more features with fewer lines of config.

Prototyping in a VM

My NAS setup needs to work every day, so I wanted to prototype my desired configuration in a VM before making changes to my system. This is not only safer, it also allows me to discover any roadblocks, and what working with NixOS feels like without making any commitments.

I copied my NixOS configuration from a previous test installation (see “How I like to install NixOS (declaratively)”) and used the following command to build a VM image and start it in QEMU:

nix build .#nixosConfigurations.storage2.config.system.build.vm

export QEMU_NET_OPTS=hostfwd=tcp::2222-:22
export QEMU_KERNEL_PARAMS=console=ttyS0
./result/bin/run-nixplay-vm

The configuration instructions below can be tried out in this VM, and once you’re happy enough with what you have, you can repeat the steps on the actual machine to migrate.

Migrating

For the migration of my actual system, I defined the following milestones that should be achievable within a typical session of about an hour (after prototyping them in a VM):

M1. Install NixOS
M2. Set up remote disk unlock
M3. Set up Samba for access
M4. Set up SSH/rsync for backups
Everything extra is nice-to-have and could be deferred to a future session on another day.

In practice, this worked out exactly as planned: the actual installation of NixOS and setting up my config to milestone M4 took a little over one hour. All the other nice-to-haves were done over the following days and weeks as time permitted.

Tip: After losing data due to an installer bug in the 2000s, I have adopted the habit of physically disconnecting all data disks (= pulling out the SATA cable) when re-installing the system disk.

M1. Install NixOS

After following “How I like to install NixOS (declaratively)”, this is my initial configuration.nix:

{ modulesPath, lib, pkgs, ... }:

{
  imports =
    [
      (modulesPath + "/installer/scan/not-detected.nix")
      ./hardware-configuration.nix
      ./disk-config.nix
    ];

  # Adding michael as trusted user means
  # we can upgrade the system via SSH (see Makefile).
  nix.settings.trusted-users = [ "michael" "root" ];
  # Clean the Nix store every week.
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 7d";
  };

  boot.loader.systemd-boot = {
    enable = true;
    configurationLimit = 10;
  };
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "storage2";
  time.timeZone = "Europe/Zurich";

  # Use systemd for networking
  services.resolved.enable = true;
  networking.useDHCP = false;
  systemd.network.enable = true;

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
  };

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  users.mutableUsers = false;
  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "networkmanager" "wheel" ];
    initialPassword = "secret";  # XXX: change!
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
  ];

  programs.zsh.enable = true;

  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

All following sections describe changes within this configuration.nix.

All devices in my home network obtain their IP address via DHCP. If I want to make an IP address static, I configure it accordingly on my router.

My NAS PCs have one specialty with regards to IP addressing: They are reachable via IPv4 and IPv6, and the IPv6 address can be derived from the IPv4 address.

Hence, I changed the systemd-networkd configuration from above such that it configures a static IPv6 address in a dynamically configured IPv6 network:

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
    ipv6AcceptRAConfig = {
      Token = "::10:0:0:252";
    };
  };

✅ This fulfills milestone M1.

M2. Set up remote disk unlock

To unlock my encrypted disks on boot, I have a custom systemd service unit that uses wget(1) and cryptsetup(8) to split the key file between the NAS and a remote server (= an attacker needs both pieces to unlock).

With CoreOS/Flatcar, my cloud-init configuration looked as follows:

coreos:
  units:
    - name: unlock.service
      command: start
      content: |
        [Unit]
        Description=unlock hard drive
        Wants=network.target
        After=systemd-networkd-wait-online.service
        Before=samba.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # Wait until the host is actually reachable.
        ExecStart=/bin/sh -c "c=0; while [ $c -lt 5 ]; do /bin/ping6 -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"
        ExecStart=/bin/sh -c "vgchange -ay"
        ExecStart=/bin/mount /dev/mapper/data-data /srv

write_files:
  - path: /etc/ssl/certs/r.zekjur.net.crt
    content: |
      -----BEGIN CERTIFICATE-----
      MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
      […]
      -----END CERTIFICATE-----

I converted it into the following NixOS configuration:

  systemd.services.unlock = {
    wantedBy = [ "multi-user.target" ];
    description = "unlock hard drive";
    wants = [ "network.target" ];
    after = [ "systemd-networkd-wait-online.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = "yes";
      ExecStart = [
        # Wait until the host is actually reachable.
        ''/bin/sh -c "c=0; while [ $c -lt 5 ]; do ${pkgs.iputils}/bin/ping -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdb2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdc2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"''

        ''/bin/sh -c "${pkgs.lvm2}/bin/vgchange -ay"''
        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
      ];
    };

  };

We’ll also need to store the custom TLS certificate file on disk. For that, we can use the environment. configuration:

  environment.etc."ssl/certs/r.zekjur.net.crt".text = ''
-----BEGIN CERTIFICATE-----
MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
[…]
-----END CERTIFICATE-----
'';

The references like ${pkgs.wget} will be replaced with a path to the Nix store (→ nix.dev documentation). On CoreOS/Flatcar, I was limited to using just the (minimal set of) software included in the base image, or I had to reach for Docker. On NixOS, we can use all packages available in nixpkgs.

After deploying and rebooting, I can access my unlocked disk under /srv! 🎉

% df -h /srv
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/data-data   15T   14T  342G  98% /srv

When listing my files, I noticed that the group id was different between my old system and the new system. This can be fixed by explicitly specifying the desired group id:

  users.groups.michael = {
    gid = 1000;  # for consistency with storage3
  };

✅ M2 is complete.

M3. Set up Samba for access

Whereas I want to configure remote disk unlock at the systemd service level, for Samba I want to use Docker: I wanted to first transfer my old (working) Docker-based setups as they are, and only later convert them to Nix.

We enable the Docker NixOS module which sets up the daemons that Docker needs and whatever else is needed to make it work:

  virtualisation.docker.enable = true;

This is already sufficient for other services to use Docker, but I also want to be able to run the docker command interactively for debugging. Therefore, I added docker to systemPackages:

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    docker
  ];

After deploying this configuration, I can run docker run -ti debian to verify things work.

The cloud-init version of samba looked like this:

coreos:
  units:
    - name: samba.service
      command: start
      content: |
        [Unit]
        Description=samba server
        After=docker.service unlock.mount
        Requires=docker.service unlock.mount

        [Service]
        Restart=always
        StartLimitInterval=0

        # Always pull the latest version (bleeding edge).
        ExecStartPre=-/usr/bin/docker pull stapelberg/docker-samba:latest

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ExecStartPre=-/usr/bin/docker kill smb
        ExecStartPre=-/usr/bin/docker rm smb
        ExecStartPre=-/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf'
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared
        ExecStartPre=/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared

        ExecStart=/usr/bin/docker run \
          -p 137:137 \
          -p 138:138 \
          -p 139:139 \
          -p 445:445 \
          --tmpfs=/run \
          -v /srv/data:/srv/data \
          --name smb \
          -t \
          smb-prepared \
            /usr/sbin/smbd --foreground --debug-stdout --no-process-group

We can translate this 1:1 to NixOS:

  systemd.services.samba = {
    wantedBy = [ "multi-user.target" ];
    description = "samba server";
    after = [ "unlock.service" ];
    requires = [ "unlock.service" ];
    serviceConfig = {
      Restart = "always";
      StartLimitInterval = 0;
      ExecStartPre = [
        # Always pull the latest version.
        ''-${pkgs.docker}/bin/docker pull stapelberg/docker-samba:latest''

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ''-${pkgs.docker}/bin/docker kill smb''
        ''-${pkgs.docker}/bin/docker rm smb''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf' ''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
      ];

      ExecStart = ''-${pkgs.docker}/bin/docker run \
           -p 137:137 \
           -p 138:138 \
           -p 139:139 \
           -p 445:445 \
           --tmpfs=/run \
           -v /srv/data:/srv/data \
           --name smb \
           -t \
           smb-prepared \
             /usr/sbin/smbd --foreground --debug-stdout --no-process-group
             '';
    };
  };
}

✅ Now I can manage my files over the network, which completes M3!

M4. Set up SSH/rsync for backups

For backing up data, I use rsync over SSH. I restrict this SSH access to run only rsync commands by using rrsync (in a Docker container). To configure the SSH authorized_keys(5) , we set:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  };

✅ A successful test backup run completes milestone M4!

Nice-to-haves

N1. Prometheus Node Exporter

I like to monitor all my machines with Prometheus (and Grafana). For network connectivity and authentication, I use the Tailscale mesh VPN.

To install Tailscale, I enable its NixOS module and make the tailscale command available:

  services.tailscale.enable = true;
  environment.systemPackages = with pkgs; [ tailscale ];

After deploying, I run sudo tailscale up and open the login link in my browser.

The Prometheus Node Exporter can also easily be enabled through its NixOS module:

  services.prometheus.exporters.node = {
    enable = true;
    listenAddress = "storage2.example.ts.net";
  };

However, this isn’t reliable yet: When Tailscale’s startup takes a while during system boot, the Node Exporter might burn through its entire restart budget when it cannot listen on the Tailscale IP address yet. We can enable indefinite restarts for the service to eventually come up:

  systemd.services."prometheus-node-exporter" = {
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      RestartSec = 1;
    };
  };

N2. Reliable mounting

While migrating my setup, I noticed that calling mount(8) from unlock.service directly is not reliable, and it’s better to let systemd manage the mounting:

  fileSystems."/srv" = {
    device = "/dev/mapper/data-data";
    fsType = "ext4";
    options = [
      "nofail"
      "x-systemd.requires=unlock.service"
    ];
  };

Afterwards, I could just remove the mount(8) call from unlock.service:

@@ -247,7 +247,10 @@ fry/U6A=
         ''/bin/sh -c "${pkgs.lvm2.bin}/bin/vgchange -ay"''
-        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
+        # Let systemd mount /srv based on the fileSystems./srv
+        # declaration to prevent race conditions: mount
+        # might not succeed while the fsck is still in progress,
+        # for example, which otherwise makes unlock.service fail.
       ];
     };

In systemd services, I can now depend on the /srv mount unit:

  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N3. nginx-healthz

To save power, I turn off my NAS when they are not in use.

My backup orchestration uses Wake-on-LAN to start a wakeup and needs to wait until the NAS is fully booted up and has mounted its /srv mount before it can start backup jobs.

For this purpose, I have configured a web server (without any files) that depends on the /srv mount. So, once the web server responds to HTTP requests, we know /srv is mounted.

The cloud-init config looked as follows:

coreos:
  units:
    - name: healthz.service
      command: start
      content: |
        [Unit]
        Description=nginx for /srv health check
        Wants=network.target
        After=srv.mount
        Requires=srv.mount
        StartLimitInterval=0

        [Service]
        Restart=always
        ExecStartPre=/bin/sh -c 'systemctl is-active docker.service'
        ExecStartPre=/usr/bin/docker pull nginx:1
        ExecStartPre=-/usr/bin/docker kill nginx-healthz
        ExecStartPre=-/usr/bin/docker rm -f nginx-healthz
        ExecStart=/usr/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1

The Docker version (ported from Flatcar Linux) looks like this:

  systemd.services.healthz = {
    description = "nginx for /srv health check";
    wants = [ "network.target" ];
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''/bin/sh -c 'systemctl is-active docker.service' ''
        ''-${pkgs.docker}/bin/docker pull nginx:1''
        ''-${pkgs.docker}/bin/docker kill nginx-healthz''
        ''-${pkgs.docker}/bin/docker rm -f nginx-healthz''
      ];

      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1
        ''
      ];
    };
  };

This configuration gets a lot simpler when migrating it from Docker to NixOS:

  # Signal readiness on HTTP port 8200 once /srv is mounted:
  networking.firewall.allowedTCPPorts = [ 8200 ];
  services.caddy = {
    enable = true;
    virtualHosts."http://10.0.0.252:8200".extraConfig = ''
      respond "ok"
    '';
  };
  systemd.services.caddy = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N4. NixOS Jellyfin

The Docker version (ported from Flatcar Linux) looks like this:

  networking.firewall.allowedTCPPorts = [ 4414 8096 ];
  systemd.services.jellyfin = {
    wantedBy = [ "multi-user.target" ];
    description = "jellyfin";
    after = [ "docker.service" "srv.mount" ];
    requires = [ "docker.service" "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''-${pkgs.docker}/bin/docker pull lscr.io/linuxserver/jellyfin:latest''
        ''-${pkgs.docker}/bin/docker rm jellyfin''
      ];
      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
          --rm \
          --net=host \
          --name=jellyfin \
          -e TZ=Europe/Zurich \
          -v /srv/jellyfin/config:/config \
          -v /srv/data/movies:/data/movies:ro \
          -v /srv/data/series:/data/series:ro \
          -v /srv/data/mp3:/data/mp3:ro \
          lscr.io/linuxserver/jellyfin:latest
        ''
      ];
    };
  };

As before, when using jellyfin from NixOS, the configuration gets simpler:

  services.jellyfin = {
    enable = true;
    openFirewall = true;
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

For a while, I had also set up compatibility symlinks that map the old location (/data/movies, inside the Docker container) to the new location (/srv/data/movies), but I encountered strange issues in Jellyfin and ended up just re-initializing my whole Jellyfin state. While the required configuration had more lines, I found it neat to move it into its own file, so here is how to do that:

Remove the lines above from configuration.nix and move them into jellyfin.nix:

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}:

{
  services.jellyfin = {
    enable = true;
    openFirewall = true;
    dataDir = "/srv/jellyfin";
    cacheDir = "/srv/jellyfin/config/cache";
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };
}

Then, in configuration.nix, add jellyfin.nix to the imports:

   imports = [
     ./hardware-configuration.nix
     ./jellyfin.nix
   ];

N5. NixOS samba

To use Samba from NixOS, I replaced my systemd.services.samba config from M3 with this:

  services.samba = {
    enable = true;
    openFirewall = true;
    settings = {
      "global" = {
        "map to guest" = "bad user";
      };
      "data" = {
        "path" = "/srv/data";
        "comment" = "public data";
        "read only" = "no";
        "create mask" = "0775";
        "directory mask" = "0775";
        "guest ok" = "yes";
      };
    };
  };
  system.activationScripts.samba_user_create = ''
      smb_password="secret"
      echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s michael
    '';

Note: Setting the samba password in the activation script works for small setups, but if you want to keep your samba passwords out of the Nix store, you’ll need to use a different approach. On a different machine, I use sops-nix to manage secrets and found that refactoring the smbpasswd call like so works reliably:

let
  setPasswords = pkgs.writeShellScript "samba-set-passwords" ''
    set -euo pipefail
    for user in michael; do
        smb_password="$(cat /run/secrets/samba_passwords/$user)"
        echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s $user
    done
  '';
in
 {
  # …
  services.samba = {
    # …as above…
  }

  systemd.services.samba-smbd.serviceConfig.ExecStartPre = [
    "${setPasswords}"
  ];

  sops.secrets."samba_passwords/michael" = {
    restartUnits = [ "samba-smbd.service" ];
  };
}

I also noticed that NixOS does not create a group for each user by default, but I am used to managing my permissions like that. We can easily declare a group like so:

  users.groups.michael = {
    gid = 1000; # for consistency with storage3
  };
  users.users.michael = {
    extraGroups = [
      "wheel" # Enable ‘sudo’ for the user.
      "docker"
      # By default, NixOS does not add users to their own group:
      # https://github.com/NixOS/nixpkgs/issues/198296
      "michael"
    ];
  };

N6. NixOS rrsync

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

To use rrsync from NixOS, I changed the configuration like so:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.rrsync}/bin/rrsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

N7. sync.pl script

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/data:/srv/data -v /root/.ssh:/root/.ssh:ro -v /etc/ssh:/etc/ssh:ro -v /etc/static/ssh:/etc/static/ssh:ro -v /nix/store:/nix/store:ro stapelberg/docker-sync",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

I wanted to stop managing the following Dockerfile to ship sync.pl:

FROM debian:stable

# Install full perl for Data::Dumper
RUN apt-get update \
    && apt-get install -y rsync ssh perl

ADD sync.pl /usr/bin/

ENTRYPOINT ["/usr/bin/sync.pl"]

To get rid of the Docker container, I translated the sync.pl file into a Nix expression that writes the sync.pl Perl script to the Nix store:

{ pkgs }:

# For string literal escaping rules (''${), see:
# https://nix.dev/manual/nix/2.26/language/string-literals#string-literals

# For writers.writePerlBin, see https://wiki.nixos.org/wiki/Nix-writers

pkgs.writers.writePerlBin "syncpl" { libraries = []; } ''
# This script is run via ssh from dornröschen.
use strict;
use warnings;
use Data::Dumper;

if (my ($destination) = ($ENV{SSH_ORIGINAL_COMMAND} =~ /^([a-z0-9.]+)$/)) {
    print STDERR "rsync version: " . `${pkgs.rsync}/bin/rsync --version` . "\n\n";
    my @rsync = (
        "${pkgs.rsync}/bin/rsync",
        "-e",
        "ssh",
        "--max-delete=-1",
        "--verbose",
        "--stats",
        # Intentionally not setting -X for my data sync,
        # where there are no full system backups; mostly media files.
        "-ax",
        "--ignore-existing",
        "--omit-dir-times",
        "/srv/data/",
        "''${destination}:/",
    );
    print STDERR "running: " . Dumper(\@rsync) . "\n";
    exec @rsync;
} else {
    print STDERR "Could not parse SSH_ORIGINAL_COMMAND.\n";
}
''

I can then reference this file by importing it in my configuration.nix and pointing it to the pkgs expression of my NixOS configuration:

{ modulesPath, lib, pkgs, ... }:

let
  syncpl = import ./syncpl.nix { pkgs = pkgs; };
in {
  imports = [ ./hardware-configuration.nix ];

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${syncpl}/bin/syncpl",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

  # For interactive usage (when debugging):
  environment.systemPackages = [ syncpl ];

  # …
}

This works, but is it the best approach? Here are some thoughts:

By managing this script in a Nix expression, I can no longer use my editor’s Perl support.
- I could probably also keep sync.pl as a separate file and use string interpolation in my Nix expression to inject an absolute path to the rsync binary into the script.
Another alternative would be to add a wrapper script to my Nix expression which ensures that $PATH contains rsync and then the script wouldn’t need an absolute path anymore.
For small glue scripts like this one, I consider it easier to manage the contents “inline” in the Nix expression, because it means one fewer file in my config directory.

N8. Sharing configs

I want to configure all my NixOS systems such that my user settings are identical everywhere.

To achieve that, I can extract parts of my configuration.nix into a user-settings.nix and then declare an accompanying flake.nix that provides this expression as an output.

After publishing these files in a git repository, I can reference said repository in my flake.nix:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    stapelbergnix.url = "github:stapelberg/nix";
  };

  outputs =
    {
      self,
      nixpkgs,
	  stapelbergnix,
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.storage2 = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          ./configuration.nix
          stapelbergnix.lib.userSettings
          # Not on this machine; We have our own networking config:
          # stapelbergnix.lib.systemdNetwork
          # Use systemd-boot as bootloader
          stapelbergnix.lib.systemdBoot
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Everything declared in the user-settings.nix can now be removed from configuration.nix!

N9. Trying immich!

One of the motivating reasons for switching away from CoreOS/Flatcar was that I couldn’t try Immich, so let’s give it a shot on NixOS:

  services.immich = {
    enable = true;
    host = "10.0.0.252";
    port = 2283;
    openFirewall = true;
    mediaLocation = "/srv/immich";
  };

  # Because /srv is a separate file system, we need to declare:
  systemd.services."immich-server" = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

Conclusion

You can find the full configuration directory on GitHub.

I am pretty happy with this NixOS setup! Previously (with CoreOS/Flatcar), I could declaratively manage my base system, but had to manage tons of Docker containers in addition. With NixOS, I can declaratively manage everything (or as much as makes sense).

Custom configuration like my SSH+rsync-based backup infrastructure can be expressed cleanly, in one place, and structured at the desired level of abstraction/reuse.

If you’re considering managing at least one other system with NixOS, I would recommend it! One of my follow-up projects is to convert storage3 (my other NAS build) from Ubuntu Server to NixOS as well to cut down on manual management. Being able to just copy the entire config to set up another system, or try out an idea in a throwaway VM, is just such a nice workflow 🥰

…but if you have just a single system to manage, probably all of this is too complicated.

How I like to install NixOS (declaratively)

2025-06-01T08:20:43+02:00

Introduction: Declarative?

The term declarative means that you describe what should be accomplished, not how. For NixOS, that means you declare what software you want your system to include (add to config option environment.systemPackages, or enable a module) instead of, say, running apt install.

A nice property of the declarative approach is that your system follows your configuration, so by reverting a configuration change, you can cleanly revert the change to the system as well.

I like to manage declarative configuration files under version control, typically with Git.

When I originally set up my current network storage build, I chose CoreOS (later Flatcar Container Linux) because it was an auto-updating base system with a declarative cloud-init config.

Ways of installing NixOS

Graphical Installer: Only for Desktops

The NixOS manual’s “Installation” section describes a graphical installer (“for desktop users”, based on the Calamares system installer and added in 2022) and a manual installer.

With the graphical installer, it’s easy to install NixOS to disk: just confirm the defaults often enough and you’ll end up with a working system. But there are some downsides:

You need to manually enable SSH after the installation — locally, not via the network.
The graphical installer generates an initial NixOS configuration for you, but there is no way to inject your own initial NixOS configuration.

The graphical installer is clearly not meant for remote installation or automated installation.

Manual Installation

The manual installer on the other hand is too manual for my taste: expand “Example 2” and “Example 3” in the NixOS manual’s Installation summary section to get an impression. To be clear, the steps are very doable, but I don’t want to install a system this way in a hurry. For one, manual procedures are prone to mistakes under stress. And also, copy & pasting commands interactively is literally the opposite of writing declarative configuration files.

Network Installation: nixos-anywhere

Ideally, I would want to perform most of the installation from the comfort of my own PC, meaning the installer must be usable over the network. Also, I want the machine to come up with a working initial NixOS configuration immediately after installation (no manual steps!).

Luckily, there is a (community-provided) solution: nixos-anywhere. You take care of booting a NixOS installer, then run a single command and nixos-anywhere will SSH into that installer, partition your disk(s) and install NixOS to disk. Notably, nixos-anywhere is configured declaratively, so you can repeat this step any time.

(I know that nixos-anywhere can even SSH into arbitrary systems and kexec-reboot them into a NixOS installer, which is certainly a cool party trick, but I like the approach of explicitly booting an installer better as it seems less risky and more generally applicable/repeatable to me.)

Setup: Installing Nix

I want to use NixOS for one of my machines, but not (currently) on my main desktop PC.

Hence, I installed only the nix tool (for building, even without running NixOS) on Arch Linux:

% sudo pacman -S nix
% sudo groupadd -r nixbld
% for n in $(seq 1 24); do sudo useradd -c "Nix build user $n" \
    -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(which nologin)" \
    nixbld$n; done
% sudo systemctl enable --now nix-daemon.socket

Now, running nix-shell -p hello should drop you in a new shell in which the GNU hello package is installed:

% export NIX_PATH=nixpkgs=channel:nixos-25.05
% nix-shell -p hello
hello

[nix-shell:/tmp]$ hello
Hello, world!

By the way, the Nix page on the Arch Linux wiki explains how to use nix to install packages, but that’s not what I am interested in: I only want to remotely manage NixOS systems.

Building your own installer

Previously, I said “you take care of booting a NixOS installer”, and that’s easy enough: write the ISO image to a USB stick and boot your machine from it (or select the ISO and boot your VM).

But before we can log in remotely via SSH, we need to manually set a password. I also need to SSH with the TERM=xterm environment variable because the termcap file of rxvt-unicode (my preferred terminal) is not included in the default NixOS installer environment. Similarly, my configured locales do not work and my preferred shell (Zsh) is not available.

Wouldn’t it be much nicer if the installer was pre-configured with a convenient environment?

With other Linux distributions, like Debian, Fedora or Arch Linux, I wouldn’t attempt to re-build an official installer ISO image. I’m sure their processes and tooling work well, but I am also sure it’s one extra thing I would need to learn, debug and maintain.

But building a NixOS installer is very similar to configuring a regular NixOS system: same configuration, same build tool. The procedure is documented in the official NixOS wiki.

I copied the customizations I would typically put into configuration.nix, imported the installation-cd-minimal.nix module from nixpkgs and put the result in the iso.nix file:

{ config, pkgs, ... }:

{
  imports = [
    
    
  ];

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "wheel" ];
    initialPassword = "SGZ3odMZIesxTuh2Y2pUaJA";  # random for this post
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    rxvt-unicode  # for terminfo
    lshw
  ];

  programs.zsh.enable = true;
  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

To build the ISO image, I set the NIX_PATH environment variable to point nix-build(1) to the iso.nix file and to select the upstream channel for NixOS 25.05:

% export NIX_PATH=nixos-config=$PWD/iso.nix:nixpkgs=channel:nixos-25.05
% nix-build '' -A config.system.build.isoImage

After about 1.5 minutes on my 2025 high-end Linux PC, the installer ISO can be found in result/iso/nixos-minimal-25.05.802216.55d1f923c480-x86_64-linux.iso (1.46 GB in size in my case).

Enabling Nix Flakes

Unfortunately, the nix project has not yet managed to enable the “experimental” new command-line interface (CLI) by default yet, despite 5+ years of being available, so we need to create a config file and enable the modern nix-command interface:

% mkdir -p ~/.config/nix
% echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

How can you tell old from new? The old commands are hyphenated (nix-build), the new ones are separated by a blank space (nix build).

You’ll notice I also enabled Nix flakes, which I use so that my nix builds are hermetic and pinned to a certain revision of nixpkgs and any other nix modules I want to include in my build. I like to compare flakes to version lock file in other programming environments: the idea is that building the system in 5 months will yield the same result as it does today.

To verify that flakes work, run nix shell (not nix-shell):

% nix shell nixpkgs#hello
/tmp 2 % hello
Hello, world!

(Re-)Installation Steps

For reference, here is the configuration I use to create a new VM for NixOS in Proxmox. The most important setting is bios=ovmf (= UEFI boot, which is not the default), so that I can use the same boot loader configuration on physical machines as in VMs:

Before we can boot our (unsigned) installer, we need to enter the UEFI setup and disable Secure Boot. Note that Proxmox enables Secure Boot by default, for example.

Then, boot the custom installer ISO on the target system, and ensure ssh michael@nixos.lan works without prompting for a password.

Declare a flake.nix with the following content:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

    disko.url = "github:nix-community/disko";
    # Use the same version as nixpkgs
    disko.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs =
    {
      nixpkgs,
      disko,
      ...
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.zammadn = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          disko.nixosModules.disko
          ./configuration.nix
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Declare your disk config in disk-config.nix:

disk-config.nix

My 2025 high-end Linux PC 🐧

2025-05-15T15:44:24+02:00

Turns out my previous attempt at this build had a faulty CPU! With the CPU replaced, the machine now is stable and fast! 🚀 In this article, I’ll go into a lot more detail about the component selection, but in a nutshell, I picked an Intel 285K CPU for low idle power, chose a 4TB SSD so I don’t have to worry about running out of storage quickly, and a capable nvidia graphics card to drive my Dell UP3218K 8K monitor.

Components

Which components did I pick for this build? Here’s the full list:

Price	Type	Article
140 CHF	Case	Fractal Define 7 Compact Black Solid
155 CHF	Power Supply	Corsair RM850x
233 CHF	Mainboard	ASUS PRIME Z890-P
620 CHF	CPU	Intel Core Ultra 9 285K
120 CHF	CPU fan	Noctua NH-D15 G2
39 CHF	Case fan	Noctua NF-A14 PWM (140 mm)
209 CHF	RAM	64 GB DDR5-6400 Corsair Vengeance (2 x 32GB)
280 CHF	Disk	4000 GB Samsung 990 Pro
554 CHF	GPU	MSI GeForce RTX 3060 Ti GAMING X TRIO

Total: 2350 CHF

…and the next couple of sections go into detail on how I selected these components.

Case

I have been a fan of Fractal cases for a couple of generations. In particular, I realized that the “Compact” series offers plenty of space even for large graphics cards and CPU coolers, so that’s now my go-to case: the Fractal Define 7 Compact (Black Solid).

My general requirements for a PC case are as follows:

No extra effort should be required for the case to be as quiet as possible.
The case should not have any sharp corners (no danger of injury!).
The case should provide just enough space for easy access to your components.
The more support the case has to encourage clean cable routing, the better.
USB3 front panel headers should be included.

I really like building components into the case and working with the case. There are no sharp edges, the mechanisms are a pleasure to use and the cable-management is well thought-out.

The only thing that wasn’t top-notch is that Fractal ships the case screws in sealed plastic packages that you need to cut open. I would have wished for a re-sealable plastic baggie so that one can keep the unused screws instead of losing them.

With this build, I have standardized all my PCs into Fractal Define 7 Compact Black cases!

Power Supply

I wanted to keep my options open regarding upgrading to an nvidia 50xx series graphics card at a later point. Those models have a TGP (“Total Graphics Power”) of 575 watts, so I needed a power supply that delivers enough power for the whole system even at peak power usage in all dimensions.

I ended up selecting the Corsair RM850x, which reviews favorably (“leader in the 850W gold category”) and was available at my electronics store of choice.

This was a good choice: the PSU indeed runs quiet, and I really like the power cables (e.g. the GPU cable) that they include: they are very flexible, which makes them easy to cable-manage.

One interesting realization was that it’s more convenient to not use the PSU’s 12VHPWR cable, but instead stick to the older 8-pin power connectors for the GPU in combination with a 12VHPWR-to-8-pin adapter. The reason is that the 12VHPWR connector’s locking mechanism is very hard to unlock, so when swapping out the GPU (as I had to do a number of times while trouble-shooting), unlocking an 8-pin connector is much easier…

SSD disk

I have been avoiding PCIe 5 SSDs so far because they consume a lot more power compared to PCIe 4 SSDs. While bulk streaming data transfer rates are higher on PCIe 5 SSDs, random transfers are not significantly faster. Most of my compute workload are random transfers, not large bulk transfers.

The power draw situation with PCIe 5 SSDs seems to be getting better lately, with the Phison E31T being the first controller that implements power saving. A disk that uses the E31T controller is the Corsair Force Series MP700 Elite. Unfortunately, said disk was unavailable when I ordered.

Instead, I picked the Samsung 990 Pro with 4 TB. I have had good experiences with the Samsung Pro series over the years (never had one die or degrade performance), and my previous 2 TB disk was starting to fill up, so the extra storage space is appreciated.

Onboard 2.5GbE Network Card

One annoying realization is that most mainboard vendors seem to have moved to 2.5 GbE (= 2.5 Gbit/s ethernet) onboard network cards. I would have been perfectly happy to play it safe and buy another Intel I225 1 GbE network card, as long as it just works with Linux.

In the 2.5 GbE space, the main players seem to be Realtek and Intel. Most mainboard vendors opted for Realtek as far as I could see.

Linux includes the r8169 driver for Realtek network cards, but whether the card will work out of the box depends on the exact revision of the network card! For example:

The AsRock Z890 Pro-A has rev 8125B. lshw: firmware=rtl8125b-2_0.0.2 07/13/20
The ASUS PRIME Z890-P has rev 8125D. lshw: firmware=rtl8125d-1_0.0.7 10/15/24

For revision 8125D, you need a recent-enough Linux version (6.13+) that includes commit “r8169: add support for RTL8125D”, accompanied by a recent-enough linux-firmware package.

Even with the latest firmware, there is some concern around stability and ASPM support. See for example this ServerFault post by someone working on the r8169 driver. But, despite the Intel 1 GbE options being well-supported at this point, Intel’s 2.5 GbE options might not fare any better than the Realtek ones: I found reports of instability with Intel’s 2.5 GbE network cards.

That said, aside from the annoying firmware requirements, the Realtek 2.5 GbE card seems to work fine for me in practice.

Mainboard

Despite the suboptimal network card choice, I decided to stick to the ASUS PRIME series of mainboards, as I made good experiences with those in my past few builds. Here are a couple of thoughts on the ASUS PRIME Z890-P mainboard I went with:

I like the quick-release PCIe mechanism: ASUS understood that people had trouble unlocking large graphics cards from their PCIe slot, so they added a lever-like mechanism that is easily reachable. In my couple of usages, this worked pretty well!
I wrote about slow boot times with my 2022 PC build that were caused by time-consuming memory training. On this ASUS board, I noticed that the board blinks the Power LED to signal that memory training is in progress. Very nice! It hadn’t occurred to me previously that the various phases of the boot could be signaled by different Power LED blinking patterns :)
- The downside of this feature is: While the machine is in suspend-to-RAM, the Power LED also blinks! This is annoying, so I might just disconnect the Power LED entirely.
The UEFI firmware includes what they call a Q-Dashboard: An overview of what is installed/connected in which slot. Quite nice:

One surprising difference between the two mainboards I tested was that the AsRock Z890 Pro-A does not seem to report the correct DIMM clock in lshw, whereas the ASUS does:

--- lshw-intel-285k-asrock.txt	2025-04-30 20:35:24 +0200
+++ lshw-intel-285k-asus.txt		2025-04-30 21:39:52 +0200
      *-firmware
           description: BIOS
-          vendor: American Megatrends International, LLC.
+          vendor: American Megatrends Inc.
           physical id: 0
-          version: 2.25
-          date: 03/24/2025
+          version: 1601
+          date: 02/07/2025
           size: 64KiB
-          capacity: 32MiB
+          capacity: 16MiB
           capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd acpi biosbootspecification uefi
[…]
      *-memory
           description: System Memory
-          physical id: 9
+          physical id: e
           slot: System board or motherboard
           size: 64GiB
         *-bank:0
-             description: DIMM [empty]
+             description: [empty]
              physical id: 0
              slot: Controller0-ChannelA-DIMM0
         *-bank:1
-             description: DIMM Synchronous 4800 MHz (0,2 ns)
+             description: DIMM Synchronous 6400 MHz (0,2 ns)
              product: CMK64GX5M2B6400C32
              vendor: Corsair
              physical id: 1
@@ -40,13 +42,13 @@
              slot: Controller0-ChannelA-DIMM1
              size: 32GiB
              width: 64 bits
-             clock: 505MHz (2.0ns)
+             clock: 2105MHz (0.5ns)
         *-bank:2
-             description: DIMM [empty]
+             description: [empty]
              physical id: 2
              slot: Controller0-ChannelB-DIMM0
         *-bank:3
-             description: DIMM Synchronous 4800 MHz (0,2 ns)
+             description: DIMM Synchronous 6400 MHz (0,2 ns)
              product: CMK64GX5M2B6400C32
              vendor: Corsair
              physical id: 3
@@ -54,7 +56,7 @@
              slot: Controller0-ChannelB-DIMM1
              size: 32GiB
              width: 64 bits
-             clock: 505MHz (2.0ns)
+             clock: 2105MHz (0.5ns)
[…]

I haven’t checked if there are measurable performance differences (e.g. if the XMP profile is truly active), but at least you now know to not necessarily trust what lshw can show you.

CPU fan

I am a long-time fan of Noctua’s products: This company makes silent fans with great cooling capacity that work reliably! For many years, I have swapped out all the fans of each of my PCs with Noctua fans, and it was always an upgrade. Highly recommended.

Hence, it is no question that I picked the latest and greatest Noctua CPU cooler for this build: the Noctua NH-D15 G2. There are a couple of things to pay attention to with this cooler:

I decided to configure it with one fan instead of two fans: Using only one fan will be the quietest setup, yet still have plenty of cooling capacity for this setup.
There are 3 different versions that differ in how their base plate is shaped. Noctua recommends: “For LGA1851, we generally recommend the regular standard version with medium base convexity” (https://noctua.at/en/intel-lga1851-all-you-need-to-know)
With a height of 168 mm, this cooler fits well into the Fractal Define 7 Compact Black.

CPU and GPU: Idle Power vs. Peak Performance

CPU choice: Intel over AMD

Probably the point that raises most questions about this build is why I selected an Intel CPU over an AMD CPU. The primary reason is that Intel CPUs are so much better at power saving!

Let me explain: Most benchmarks online are for gamers and hence measure a usage curve that goes “start game, run PC at 100% resources for hours”. Of course, when you never let the machine idle, you would care about power efficiency: how much power do you need to use to achieve the desired result?

My use-case is software development, not gaming. My usage curve oscillates between “barely any usage because Michael is reading text” to “complete this compilation as quickly as possible with all the power available”. For me, I need both absolute power consumption at idle, and absolute performance to be best-of-class.

AMD’s CPUs offer great performance (the recently released Ryzen 9 9950X3D is even faster than the Intel 9 285K), and have great power efficiency, but poor power consumption at idle: With ≈35W of idle power draw, Zen 5 CPUs consume ≈3x as much power as Intel CPUs!

Intel’s CPUs offer great performance (like AMD), but excellent power consumption at idle.

Therefore, I can’t in good conscience buy an AMD CPU, but if you want a fast gaming-only PC or run an always-loaded HPC cluster with those CPUs, definitely go ahead :)

Graphics card: nvidia over AMD

I don’t necessarily recommend any particular nvidia graphics card, but I have had to stick to nvidia cards because they are the only option that work with my picky Dell UP3218K monitor.

From time to time, I try out different graphics cards. Recently, I got myself an AMD Radeon RX 9070 because I read that it works well with open source drivers.

While the Radeon RX 9070 works with my monitor (great!), it seems to consume 45W in idle, which is much higher than my nvidia cards, which idle at ≈ 20W. This is unacceptable to me: Aside from high power costs and wasting precious resources, the high power draw also means that my room will be hotter in summer and the fans need to spin faster and therefore louder.

People asked me on Social Media if this could be a measurement error (like, the card reporting inaccurate values), so I double-checked with a myStrom WiFi Switch and confirmed that with the Radeon card, the PC indeed draws 20-30W more from the wall socket.

Why Low Idle Power is so important

In the comments for my my previous blog post about the first build of this machine not running stable, people were asking why it is worth it to optimize a few watts of power usage. People calculate what higher power usage might cost, put it in relation to the total cost of the components, and conclude that saving ±10% of the price can’t possibly be worth the effort.

Let me try to illustrate the importance of low idle power with this anecdote: For one year, I was suffering from an nvidia driver bug that meant the GPU would not clock down to the most efficient power-saving mode (because of the high resolution of my monitor). The 10-20W of difference should have been insignificant. Yet, when the bug was fixed, I noticed how my PC got quieter (fans don’t need to spin up) and my room noticeably cooled down, which was great as it was peak temperatures in summer.

To me, having a whisper-quiet computing environment that does not heat up my room is a great, actual, real-life, measurable benefit. Not wasting resources and saving a tiny amount of money is a nice cherry on top.

Obviously all the factors are very dependent on your specific situation: Your house’s thermal behavior might differ from mine, your tolerance for noise (and/or baseline noise levels) might be different, you might put more/less weight on resource usage, etc.

Installation

UEFI setup

On the internet, I read that there was some issue related to the Power Limits that mainboards come with by default. Therefore, I did a UEFI firmware update immediately after getting the mainboard. I upgraded to version 1404 (2025/01/10) using the provided ZIP file (PRIME-Z890-P-ASUS-1404.zip) on an MS-DOS FAT-formatted USB stick with the EZ Flash tool in the UEFI firmware interface. Tip: do not extract the ZIP file, otherwise the EZ Flash tool cannot update the Intel ME firmware. Just put the ZIP file onto the USB disk as-is.

I verified that with this UEFI version, the Power Limit 1 (PL1) is 250W, and ICCMAX=347A, which are exactly the values that Intel recommends. Great!

I also enabled XMP and verified that memtest86 reported no errors.

Software setup: early adopter pains

To copy over the data from the old disk to the new disk, I wanted to boot a live linux distribution (specifically, grml.org) and follow my usual procedure: boot with the old disk and the new (empty) disk, then use dd to copy the data. It’s nice and simple, hard to screw up.

Unfortunately, while grml 2024.12 technically does boot up, there are two big problems:

There is no network connectivity because the kernel and linux-firmware versions are too old.
- Kernel commit r8169: add support for RTL8125D is not included.
I could not get Xorg to work at all. Not with the Intel integrated GPU, nor with the nvidia dedicated GPU. Not with nomodeset or any of the other options in the grml menu. This wasn’t merely a convenience problem: I needed to use gparted (the graphical version) for its partition moving/resizing support.

Ultimately, it was easier to upgrade my old PC to Linux 6.13 and linux-firmware 20250109, then put in the new disk and copy over the installation.

TRIM your SSDs

SSD disks can degrade over time, so it is essential that the Operating System tells the SSD firmware about freed-up blocks (for wear leveling). When using full-disk encryption, all involved layers need to have TRIM support enabled.

I think I saw the effect of an incorrectly configured TRIM setup in action back in 2022, when I copied my data from a Force MP600 to a WD Black SN850, which unexpectedly took many hours!

To make sure my disk has a long and healthy life, I double-checked that both periodic and continuous TRIM are enabled on my Arch Linux system: The fstab(5) file contains the discard option (and mount(8) lists the discard option), and fstrim.service ran within the last week:

systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
fstrim[779617]: /boot: 10.1 GiB (10799427584 bytes) trimmed on /dev/nvme0n1p1
fstrim[779617]: /: 1.8 TiB (2018906263552 bytes) trimmed on /dev/mapper/cryptroot
systemd[1]: fstrim.service: Deactivated successfully.

Speaking of copying data: the transfer from my WD Black SN850 to my Samsung 990 PRO ran at 856 MB/s and took about 40 minutes in total.

Performance

Here are the total times for a couple of typical workloads I run:

Workload	12900K (2022)	285K (2025)
build Go 1.24.3 (`cd src; ./make.bash`)	≈35s	≈26s
gokrazy/rsync tests (`make test`)	≈0.5s	≈0.4s
gokrazy UEFI test (`go test ./integration/...`)	≈30s	≈10s
gokrazy Linux compile (`gokr-rebuild-kernel -cross=arm64`)	3m 13s	2m 7s

The performance boost is great! Building Linux kernels a whole minute faster is really nice.

Stability issues

In March, I published an article about how the first build of this machine was not stable, in which you can read in detail about the various crashes I ran into.

Now, in early May, I know for sure that the CPU was defective, after a lengthy trouble-shooting in which I swapped out all the other parts of this PC, sent back the CPU and got a new one.

The CPU was the most annoying component to diagnose in this build because it’s an LGA 1851 socket and I don’t (yet) have any other machines which uses that same socket. AMD’s approach of sticking to each socket for a longer time would have been better in this situation.

Stress testing

When I published my earlier blog post about the PC being unstable, I did not really know how to reliably trigger the issue. Some compute-intensive tasks like running a Django test suite seemed to trigger the issue. I suspect that the problem somehow got worse, because when I started stress testing the machine, suddenly it would crash every time when building a Linux kernel.

That got me curious to see if other well-known CPU stress testers like Prime95 would show problems, and indeed: within seconds, Prime95 would report errors.

I figured I would use Prime95 as a quick signal: if it reports errors, the machine is faulty. This typically happens within seconds of starting Prime95.

If Prime95 reported no errors, I would use Linux kernel compilation as a slow signal: if I can successfully build a kernel, the machine is likely stable enough.

The specific setup I used is to run ./mprime -m, hit N (do not participate in distributed computation projects), then Enter a few times to confirm the defaults. Eventually, Prime95 starts calculating, which pushes the CPU to 100% usage (see the dstat(1) -like output by my gokrazy/stat implementation) and draws the expected ≈300W of power from the wall:

In addition, I also ran MemTest86 for a few hours:

To be clear: I also successfully ran MemTest86 on the previous, unstable build, so only running MemTest86 is not good enough if you are dealing with a faulty CPU.

RMA timeline

Jan 15th: I receive the components for my new PC
- In January and February, the PC crashes occasionally.
Mar 4th: I switch back to my old PC and start writing my blog post
Mar 19th: I publish my blog post about the machine not being stable
- The online discussion does not result in any interesting tips or leads.
Mar 20th: I order the AsRock Z890 Pro-A mainboard to ensure the mainboard is OK
Mar 24th: the AsRock Z890 Pro-A arrives
Apr 5th (Sat): started an RMA for the CPU
- They ask me to send the CPU to orderflow, which is the merchant that fulfilled my order.
- Typically, I prefer buying directly at digitec, but many PC components seem to only be available from orderflow on digitec nowadays.
Apr 9th (Wed): package arrives at orderflow (digitec gave me a non-priority return label)
Apr 14th (Mon): I got the following mail from digitec’s customer support and had to explain that I have thoroughly diagnosed the CPU as defective (a link to my blog post was sufficient):

Händler hat dies beim Hersteller angemeldet und dieser hat folgende Fragen:

Um sicherzugehen, dass wir Sie richtig verstehen: Sie haben die CPU auf zwei verschiedenen Motherboards getestet und das gleiche Problem besteht weiterhin?

Könnten Sie uns mitteilen, welche Marke und welches Modell die beiden verwendeten Motherboards sind?

Wurde auf beiden Motherboards die neueste BIOS-Version verwendet?

Bestand das Problem von Anfang an oder trat es erst später auf?

Wurde der Prozessor übertaktet? (Bitte beachten Sie, dass durch Übertakten die Garantie erlischt.)

Apr 25th (Fri): orderflow hands the replacement CPU to Swiss Post
May 1st (Thu): the machine successfully passes stress tests; I start using it

In summary, I spent March without a working PC, but that was because I didn’t have much time to pursue the project. Then, I spent April without a working PC because RMA’ing an Intel CPU through digitec seems pretty slow. I would have wished for a little more trust and a replacement CPU right away.

Conclusion

What a rollercoaster and time sink this build was! I have never received a faulty-on-arrival CPU in my entire life before. How did the CPU I first received pass Intel’s quality control? Or did it pass QC, but was damaged in transport? I will probably never know.

From now on, I know to extensively stress test new PC builds for stability to detect such issues quicker. Should the CPU be faulty, unfortunately getting it replaced is a month-long process — it’s very annoying to have such a costly machine just gather dust for a month.

But, once the faulty component was replaced, this is my best PC build yet:

The case is the perfect size for the components and offers incredibly convenient access to all components throughout the entire lifecycle of this PC, including the troubleshooting period, and the later stages of its life when this PC will be rotated into its “lab machine” period before I sell it second-hand to someone who will hopefully use the machine for another few years.

The machine is quiet, draws little power (for such a powerful machine) and really packs a punch!

As usual, I run Linux on this PC and haven’t noticed any problems in my day-to-day usage. I use suspend-to-RAM multiple times a day without any issues.

I hope some of these details were interesting and useful to you in your own PC builds!

If you want to learn about which peripherals I use aside from my 8K monitor (e.g. the Kinesis Advantage keyboard, Logitech MX Ergo trackball, etc.), check out my post stapelberg uses this: my 2020 desk setup. I might publish an updated version at some point :)

In praise of grobi for auto-configuring X11 monitors

2025-05-10T08:24:00+02:00

I have recently started using the grobi program by Alexander Neumann again and was delighted to discover that it makes using my fiddly (but wonderful) Dell 32-inch 8K monitor (UP3218K) monitor much more convenient — I get a signal more quickly than with my previous, sleep-based approach.

Previously, when my PC woke up from suspend-to-RAM, there were two scenarios:

The monitor was connected. My sleep program would power on the monitor (if needed), sleep a little while and then run xrandr(1) to (hopefully) configure the monitor correctly.
The monitor was not connected, for example because it was still connected to my work PC.

In scenario ②, or if the one-shot configuration attempt in scenario ① fails, I would need to SSH in from a different computer and run xrandr manually so that the monitor would show a signal:

% DISPLAY=:0 xrandr \
  --output DP-4 --mode 3840x4320 --panning 0x0+0+0 \
  --output DP-2 --right-of DP-4 --mode 3840x4320 --panning 0x0+3840+0

Automatic monitor configuration with grobi

I have now completely solved this problem by creating the following ~/.config/grobi.conf file:

rules:
  - name: UP3218K

    outputs_connected: [DP-2, DP-4]

	# DP-4 is left, DP-2 is right
    configure_row:
        - DP-4@3840x4320
        - DP-2@3840x4320

    # atomic instructs grobi to only call xrandr once and configure all the
    # outputs. This does not always work with all graphic cards, but is
	# needed to successfully configure the UP3218K monitor.
    atomic: true

…and installing / enabling grobi (on Arch Linux) using:

% sudo pacman -S grobi
% systemctl --user enable --now grobi

Whenever grobi detects that my monitor is connected (it listens for X11 RandR output change events), it will run xrandr(1) to configure the monitor resolution and positioning.

To check what grobi is seeing/doing, you can use:

% systemctl --user status grobi
% journalctl --user -u grob

For example, on my system, I see:

grobi: 18:31:48.823765 outputs: [HDMI-0 (primary) DP-0 DP-1 DP-2 (connected) 3840x2160+ [DEL-16711-808727372-DELL UP3218K-D2HP805I043L] DP-3 DP-4 (connected) 3840x21>
grobi: 18:31:48.823783 new rule found: UP3218K
grobi: 18:31:48.823785 enable outputs: [DP-4@3840x4320 DP-2@3840x4320]
grobi: 18:31:48.823789 using one atomic call to xrandr
grobi: 18:31:48.823806 running command /usr/bin/xrandr xrandr --output DP-4 --mode 3840x4320 --output DP-2 --mode 3840x4320 --right-of DP-4
grobi: 18:31:49.285944 new RANDR change event received

Notably, the instructions for getting out of a bad state (no signal) are now to power off the monitor and power it back on again. This will result in RandR output change events, which will trigger grobi, which will run xrandr, which configures the monitor. Nice!

Why not autorandr?

No particular reason. I knew grobi.

If nothing else, grobi is written in Go, so it’s likely to keep working smoothly over the years.

Does grobi work on Wayland?

Probably not. There is no mention of Wayland over on the grobi repository.

Bonus: my Suspend-to-RAM setup

As a bonus, this section describes the other half of my monitor-related automation.

When I suspend my PC to RAM, I either want to wake it up manually later, for example by pressing a key on the keyboard or by sending a Wake-on-LAN packet, or I want it to wake up automatically each morning at 6:50 — that way, daily cron jobs have some time to run before I start using the computer.

To accomplish this, I use zleep, a wrapper program around rtcwake(8) and systemctl suspend that integrates with the myStrom switch smart plug to turn off power to the monitor entirely. This is worthwhile because the monitor draws 30W even in standby!

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"os"
	"os/exec"
	"time"
)

var (
	resume = flag.Bool("resume",
		false,
		"run resume behavior only (turn on monitor via smart plug)")

	noMonitor = flag.Bool("no_monitor",
		false,
		"disable turning off/on monitor")
)

func monitorPower(ctx context.Context, method, cmnd string) error {
	if *noMonitor {
		log.Printf("[monitor power] skipping because -no_monitor flag is set")
		return nil
	}
	log.Printf("[monitor power] command: %v", cmnd)
	u, err := url.Parse("http://myStrom-Switch-A46FD0/" + cmnd)
	if err != nil {
		return err
	}
	for {
		if err := ctx.Err(); err != nil {
			return err
		}
		req, err := http.NewRequest(method, u.String(), nil)
		if err != nil {
			return err
		}
		ctx, canc := context.WithTimeout(ctx, 5*time.Second)
		defer canc()
		req = req.WithContext(ctx)
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			log.Print(err)
			time.Sleep(1 * time.Second)
			continue
		}
		if resp.StatusCode != http.StatusOK {
			log.Printf("unexpected HTTP status code: got %v, want %v", resp.Status, http.StatusOK)
			time.Sleep(1 * time.Second)
			continue
		}
		log.Printf("[monitor power] request succeeded")
		return nil
	}
}

func nextWakeup(now time.Time) time.Time {
	midnight := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.Local)
	if now.Hour() < 6 {
		// wake up today
		return midnight.Add(6*time.Hour + 50*time.Minute)
	}

	// wake up tomorrow
	return midnight.Add(24 * time.Hour).Add(6*time.Hour + 50*time.Minute)
}

func runResume() error {
	// Retry for up to one minute to give the network some time to come up
	ctx, canc := context.WithTimeout(context.Background(), 1*time.Minute)
	defer canc()
	if err := monitorPower(ctx, "GET", "relay?state=1"); err != nil {
		log.Print(err)
	}
	return nil
}

func zleep() error {
	ctx := context.Background()

	now := time.Now().Truncate(1 * time.Second)
	wakeup := nextWakeup(now)
	log.Printf("now   : %v", now)
	log.Printf("wakeup: %v", wakeup)
	log.Printf("wakeup: %v (timestamp)", wakeup.Unix())

	// assumes hwclock is running in UTC (see timedatectl | grep local)

	// Power the monitor off in 15 seconds.
	// mode=on is intentional: https://api.mystrom.ch/#e532f952-36ea-40fb-a180-a57b835f550e
	// - the switch will be turned on (already on, so this is a no-op)
	// - the switch will wait for 15 seconds
	// - the switch will be turned off
	if err := monitorPower(ctx, "POST", "timer?mode=on&time=15"); err != nil {
		log.Print(err)
	}

	sleep := exec.Command("sh", "-c", fmt.Sprintf("sudo rtcwake -m no --verbose --utc -t %v && sudo systemctl suspend", wakeup.Unix()))
	sleep.Stdout = os.Stdout
	sleep.Stderr = os.Stderr
	fmt.Printf("running %v\n", sleep.Args)
	if err := sleep.Run(); err != nil {
		return fmt.Errorf("%v: %v", sleep.Args, err)
	}

	return nil
}

func main() {
	flag.Parse()
	if *resume {
		if err := runResume(); err != nil {
			log.Fatal(err)
		}
	} else {
		if err := zsleep(); err != nil {
			log.Fatal(err)
		}
	}
}

To turn power to the monitor on after resuming, I placed the following shell script in /lib/systemd/system-sleep/zleep.sh:

#!/bin/sh

case "$1" in
	pre)	exit 0
		;;
	post)	/usr/local/bin/zleep -resume
		exit 0
		;;
 	*)	exit 1
		;;
esac

Once power is on, grobi will detect and configure the monitor.

Here is the program in action:

2025/05/06 21:58:32 now   : 2025-05-06 21:58:32 +0200 CEST
2025/05/06 21:58:32 wakeup: 2025-05-07 06:50:00 +0200 CEST
2025/05/06 21:58:32 wakeup: 1746593400 (timestamp)
2025/05/06 21:58:32 [monitor power] command: timer?mode=on&time=15
2025/05/06 21:58:32 [monitor power] request succeeded
running [sh -c sudo rtcwake -m no --verbose --utc -t 1746593400 && sudo systemctl suspend]
Using UTC time.
	delta   = 0
	tzone   = 0
	tzname  = UTC
	systime = 1746561512, (UTC) Tue May  6 19:58:32 2025
	rtctime = 1746561512, (UTC) Tue May  6 19:58:32 2025
alarm 1746593400, sys_time 1746561512, rtc_time 1746561512, seconds 0
rtcwake: wakeup using /dev/rtc0 at Wed May  7 04:50:00 2025
suspend mode: no; leaving

Intel 9 285K on ASUS Z890: not stable!

2025-03-19T17:35:38+01:00

In January I ordered the components for a new PC and expected that I would publish a successor to my 2022 high-end Linux PC 🐧 article. Instead, I am now sitting on a PC which regularly encounters crashes of the worst-to-debug kind, so I am publishing this article as a warning for others in case you wanted to buy the same hardware.

Components

Which components did I pick for this build? Here’s the full list:

Price	Type	Article
140 CHF	Case	Fractal Define 7 Compact Black Solid
155 CHF	Power Supply	Corsair RM850x
233 CHF	Mainboard	ASUS PRIME Z890-P
620 CHF	CPU	Intel Core Ultra 9 285k
120 CHF	CPU fan	Noctua NH-D15 G2
39 CHF	Case fan	Noctua NF-A14 PWM (140 mm)
209 CHF	RAM	64 GB DDR5-6400 Corsair Vengeance (2 x 32GB)
280 CHF	Disk	4000 GB Samsung 990 Pro
940 CHF	GPU	Inno3D GeForce RTX4070 Ti

Total: ≈1800 CHF, excluding the Graphics Card I re-used from a previous build.

…and the next couple of sections go into detail on how I selected these components.

Case

I really like building components into the case and working with the case. There are no sharp edges, the mechanisms are a pleasure to use and the cable-management is well thought-out.

Power Supply

I wanted to keep my options open regarding upgrading to an nVidia 50xx series graphics card at a later point. Those models have a TGP (“Total Graphics Power”) of 575 watts, so I needed a power supply that delivers enough power for the whole system even at peak power usage in all dimensions.

I ended up selecting the Corsair RM850x, which reviews favoribly (“leader in the 850W gold category”) and was available at my electronics store of choice.

This was a good choice: the PSU indeed runs quiet, and I really like the power cables (e.g. the GPU cable) that they include: they are very flexible, which makes them easy to cable-manage.

SSD disk

Instead, I picked the Samsung 990 Pro with 4 TB. I made good experiences with the Samsung Pro series over the years (never had one die or degrade performance), and my previous 2 TB disk is starting to fill up, so the extra storage space is appreciated.

Mainboard

In the 2.5 GbE space, the main players seem to be Realtek and Intel. Most mainboard vendors opted for Realtek as far as I could see.

Linux includes the r8169 driver for Realtek network cards, but you need a recent-enough Linux version (6.13+) that includes commit “r8169: add support for RTL8125D”, accompanied by a recent-enough linux-firmware package. Even then, there is some concern around stability and ASPM support. See for example this ServerFault post by someone working on the r8169 driver.

Despite the Intel 1 GbE options being well-supported at this point, Intel’s 2.5 GbE options might not fare any better than the Realtek ones: I found reports of instability with Intel’s 2.5 GbE network cards.

Aside from the network cards, I decided to stick to the ASUS prime series of mainboards, as I made good experiences with those in my past few builds. Here are a couple of thoughts on the ASUS PRIME Z890-P mainboard I went with:

I like the quick-release PCIe mechanism: ASUS understood that people had trouble unlocking large graphics cards from their PCIe slot, so they added a lever-like mechanism that is easily reachable. In my couple of usages, this worked pretty well!
I wrote about slow boot times with my 2022 PC build that were caused by time-consuming memory training. On this ASUS board, I noticed that they blink the Power LED to signal that memory training is in progress. Very nice! It hadn’t occurred to me previously that the various phases of the boot could be signaled by different Power LED blinking patterns :)
- The downside of this feature is: While the machine is in suspend-to-RAM, the Power LED also blinks! This is annoying, so I might just disconnect the Power LED entirely.
The UEFI firmware includes what they call a Q-Dashboard: An overview of what is installed/connected in which slot. Quite nice:

CPU fan

I am a long-time fan of Noctua’s products: This company makes silent fans with great cooling capacity that work reliably! For many years, I have swapped out every of my PC’s fans with Noctua fans, and it was always an upgrade. Highly recommended.

Hence, it is no question that I picked the latest and greatest Noctua CPU cooler for this build: the Noctua NH-D15 G2. There are a couple of things to pay attention to with this cooler:

I decided to configure it with one fan instead of two fans: Using only one fan will be the quietest setup, yet still have plenty of cooling capacity for this setup.
There are 3 different versions that differ in how their base plate is shaped. Noctua recommends: “For LGA1851, we generally recommend the regular standard version with medium base convexity” (https://noctua.at/en/intel-lga1851-all-you-need-to-know)
The height of this cooler is 168 mm. This fits well into the Fractal Define 7 Compact Black.

CPU

Probably the point that raises most questions about this build is why I selected an Intel CPU over an AMD CPU. The primary reason is that Intel CPUs are so much better at power saving!

Intel’s CPUs offer great performance (like AMD), but excellent power consumption at idle.

Therefore, I can’t in good conscience buy an AMD CPU, but if you want a fast gaming-only PC or run an always-loaded HPC cluster with those CPUs, definitely go ahead :)

Graphics card

I don’t necessarily recommend any particular nVidia graphics card, but I have had to stick to nVidia cards because they are the only option that work with my picky Dell UP3218K monitor.

From time to time, I try out different graphics cards. Recently, I got myself an AMD Radeon RX 9070 because I read that it works well with open source drivers.

While the Radeon RX 9070 works with my monitor (great!), it seems to consume 45W in idle, which is much higher than my nVidia cards, which idle at ≈ 20W. This is unacceptable to me: Aside from high power costs and wasting precious resources, the high power draw also means that my room will be hotter in summer and the fans need to spin faster and therefore louder.

Maybe I’ll write a separate article about the Radeon RX 9070.

Installation

UEFI setup

On the internet, I read that there was some issue related to the Power Limits that mainboards come with by default. Therefore, I did a UEFI firmware update first thing after getting the mainboard. I upgraded to version 1404 (2025/01/10) using the provided ZIP file (PRIME-Z890-P-ASUS-1404.zip) on an MS-DOS FAT-formatted USB stick with the EZ Flash tool in the UEFI firmware interface. Tip: do not extract the ZIP file, otherwise the EZ Flash tool cannot update the Intel ME firmware. Just put the ZIP file onto the USB disk as-is.

I verified that with this UEFI version, the Power Limit 1 (PL1) is 250W, and ICCMAX=347A, which are exactly the values that Intel recommends. Great!

I also enabled XMP and verified that memtest86 reported no errors.

Software setup: early adopter pains

Unfortunately, while grml 2024.12 technically does boot up, there are two big problems:

There is no network connectivity because the kernel and linux-firmware versions are too old.
- r8169: add support for RTL8125D
I could not get Xorg to work at all. Not with the Intel integrated GPU, nor with the nVidia dedicated GPU. Not with nomodeset or any of the other options in the grml menu. This wasn’t merely a convenience problem: I needed to use gparted (the graphical version) for its partition moving/resizing support.

Ultimately, it was easier to upgrade my old PC to Linux 6.13 and linux-firmware 20250109, then put in the new disk and copy over the installation.

Stability issues

At this point (early February), I switched to this new machine as my main PC.

Unfortunately, I could never get it to run stable! This journal shows you some of the issues I faced and what I tried to troubleshoot them.

Xorg dying after resume-from-suspend

One of the first issues I encountered with this system was that after resuming from suspend-to-RAM, I was greeted with a login window instead of my X11 session. The logs say:

(EE) NVIDIA(GPU-0): Failed to acquire modesetting permission.
(EE) Fatal server error:
(EE) EnterVT failed for screen 0
(EE) 
(EE) 
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE) 
(WW) NVIDIA(0): Failed to set the display configuration
(WW) NVIDIA(0):  - Setting a mode on head 0 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 1 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 2 failed: Insufficient permissions
(WW) NVIDIA(0):  - Setting a mode on head 3 failed: Insufficient permissions
(EE) Server terminated with error (1). Closing log file.

I couldn’t find any good tips online for this error message, so I figured I’d wait and see how frequently this happens before investigating further.

Feb 18: xHCI host controller dying

On Feb 18th, after resume-from-suspend, none of my USB peripherals would work anymore! This affected all USB ports of the machine and could not be fixed, not even by a reboot, until I fully killed power to the machine! In the kernel log, I saw the following messages:

xhci_hcd 0000:80:14.0: xHCI host not responding to stop endpoint command
xhci_hcd 0000:80:14.0: xHCI host controller not responding, assume dead
xhci_hcd 0000:80:14.0: HC died; cleaning up

Feb 24: xHCI host controller dying

The HC dying issue happened again when I was writing an SD card in my USB card reader:

xhci_hcd 0000:80:14.0: HC died; cleaning up

Feb 24: → UEFI update, disable XMPP

To try and fix the host controller dying issue, I updated the UEFI firmware to version 1601 and disabled the XMPP RAM profile.

Feb 26: → switch back from GeForce 4070 Ti to 3060 Ti

To rule out any GPU-specific issues, I decided to switch back from the Inno3D GeForce RTX4070 Ti to my older MSI GeForce RTX 3060 Ti.

Feb 28: PC dying on suspend-to-RAM

On Feb 28th, my PC did not resume from suspend-to-RAM. It would not even react to a ping, I had to hard-reset the machine. When checking the syslog afterwards, there are no entries.

I checked my power monitoring and saw that the machine consumed 50W (well above idle power, and far above suspend-to-RAM power) throughout the entire night. Hence, I suspect that the suspend-to-RAM did not work correctly and the machine never actually suspended.

Mar 4th: PC dying when running django tests

On March 4th, I was running the test suite for a medium-sized Django project (= 100% CPU usage) when I encountered a really hard crash: The machine stopped working entirely, meaning all peripherals like keyboard and mouse stopped responding, and the machine even did not respond to a network ping anymore.

At this point, I had enough and switched back to my 2022 PC.

Conclusion

What use is a computer that doesn’t work? My hierarchy of needs contains stability as the foundation, then speed and convenience. This machine exhausted my tolerance for frustration with its frequent crashes.

Manawyrm actually warned me about the ASUS board:

ASUS boards are a typical gamble as always – they fired their firmware engineers about 10 years ago, so you might get a nightmare of ACPI troubleshooting hell now (or it’ll just work). ASRock is worth a look as a replacement if that happens. Electronics are usually solid, though…

I didn’t expect that this PC would crash so hard, though. Like, if it couldn’t suspend/resume that would be one thing (a dealbreaker, but somewhat expected and understandable, probably fixable), but a machine that runs into a hard-lockup when compiling/testing software? No thanks.

I will buy a different mainboard to see if that helps, likely the ASRock Z890 Pro-A. If you have any recommendations for a Z890 mainboard that actually works reliably, please let me know!

Update 2025-04-17: I have received the ASRock Z890 Pro-A, but the machine shows exactly the same symptoms! I also swapped the power supply, which also did not help. Running Prime95 crashed almost immediately. At this point, I have to assume the CPU itself is defective and have started an RMA. I will post another update once (if?) I get a replaced CPU.

Update 2025-05-11: The CPU was faulty indeed! See My 2025 high-end Linux PC for a new article on this build, now with a working CPU.

Tips to debug hanging Go programs

2025-02-27T17:51:38+01:00

I was helping someone get my gokrazy/rsync implementation set up to synchronize RPKI data (used for securing BGP routing infrastructure), when we discovered that with the right invocation, my rsync receiver would just hang indefinitely.

This was a quick problem to solve, but in the process, I realized that I should probably write down a few Go debugging tips I have come to appreciate over the years!

Scenario: hanging Go program

If you want to follow along, you can reproduce the issue by building an older version of gokrazy/rsync, just before the bug fix commit (you’ll need Go 1.22 or newer):

git clone https://github.com/gokrazy/rsync
cd rsync
git reset --hard 6c89d4dda3be055f19684c0ed56d623da458194e^
go install ./cmd/...

Now we can try to sync the repository:

% gokr-rsync \
  -rtO \
  --delete \
  rsync://rsync.paas.rpki.ripe.net/repository/ \
  /tmp/rpki-repo
[…]
2025/02/08 09:35:10 Opening TCP connection to rsync.paas.rpki.ripe.net:873
2025/02/08 09:35:10 rsync module "repo", path "repo/"
2025/02/08 09:35:10 (Client) Protocol versions: remote=31, negotiated=27
2025/02/08 09:35:10 Client checksum: md4
2025/02/08 09:35:10 sending daemon args: [--server --sender -tr . repo/]
2025/02/08 09:35:10 exclusion list sent
2025/02/08 09:35:10 receiving file list
2025/02/08 09:35:11 [Receiver] i=0 ? . mode=40755 len=4096 uid=0 gid=0 flags=?
[…]
2025/02/08 09:35:11 [Receiver] i=89 ? clonoth/1/3139332e33322e3130302e302f32342d3234203d3e203537313936.roa mode=100644 len=1747 uid=0 gid=0 flags=?

…and then the program just sits there.

Tip 1: Press Ctrl+\ (SIGQUIT) to print a stack trace

The easiest way to look at where a Go program is hanging is to press Ctrl+\ (backslash) to make the terminal send it a SIGQUIT signal. When the Go runtime receives SIGQUIT, it prints a stack trace to the terminal before exiting the process. This behavior is enabled by default and can be customized via the GOTRACEBACK environment variable, see the runtime package docs.

Here is what the output looks like in our case. I have made the font small so that you can recognize the shape of the output (the details are not important, continue reading below):

^\SIGQUIT: quit
PC=0x47664e m=0 sigcode=128

goroutine 0 gp=0x6e6020 m=0 mp=0x6e6ec0 [idle]:
internal/runtime/syscall.Syscall6()
	/home/michael/sdk/go1.23.0/src/internal/runtime/syscall/asm_linux_amd64.s:36 +0xe fp=0x7ffc58665090 sp=0x7ffc58665088 pc=0x47664e
internal/runtime/syscall.EpollWait(0x586651e0?, {0x7ffc5866511c?, 0x3000000018?, 0x7ffc586651f0?}, 0x58665110?, 0x7ffc?)
	/home/michael/sdk/go1.23.0/src/internal/runtime/syscall/syscall_linux.go:32 +0x45 fp=0x7ffc586650e0 sp=0x7ffc58665090 pc=0x4765e5
runtime.netpoll(0xc0000000c0?)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll_epoll.go:116 +0xd2 fp=0x7ffc58665768 sp=0x7ffc586650e0 pc=0x432332
runtime.findRunnable()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:3580 +0x8c5 fp=0x7ffc586658e0 sp=0x7ffc58665768 pc=0x43f045
runtime.schedule()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:3995 +0xb1 fp=0x7ffc58665918 sp=0x7ffc586658e0 pc=0x4405b1
runtime.park_m(0xc0000061c0)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:4102 +0x1eb fp=0x7ffc58665970 sp=0x7ffc58665918 pc=0x4409cb
runtime.mcall()
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:459 +0x4e fp=0x7ffc58665988 sp=0x7ffc58665970 pc=0x470e2e

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x452658?, 0x0?, 0x98?, 0xb3?, 0xb?)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:424 +0xce fp=0xc0000eb358 sp=0xc0000eb338 pc=0x46bc0e
runtime.netpollblock(0x4a01b8?, 0x4058e6?, 0x0?)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000eb390 sp=0xc0000eb358 pc=0x4318f7
internal/poll.runtime_pollWait(0x7ef586628808, 0x72)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll.go:351 +0x85 fp=0xc0000eb3b0 sp=0xc0000eb390 pc=0x46af05
internal/poll.(*pollDesc).wait(0xc0000ce180?, 0xc00020e99c?, 0x0)
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000eb3d8 sp=0xc0000eb3b0 pc=0x4b0ce7
internal/poll.(*pollDesc).waitRead(...)
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0000ce180, {0xc00020e99c, 0x4, 0x4})
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc0000eb470 sp=0xc0000eb3d8 pc=0x4b17da
net.(*netFD).Read(0xc0000ce180, {0xc00020e99c?, 0x6eeea0?, 0x1?})
	/home/michael/sdk/go1.23.0/src/net/fd_posix.go:55 +0x25 fp=0xc0000eb4b8 sp=0xc0000eb470 pc=0x4f7e85
net.(*conn).Read(0xc000206000, {0xc00020e99c?, 0xc000212000?, 0x6e6ec0?})
	/home/michael/sdk/go1.23.0/src/net/net.go:189 +0x45 fp=0xc0000eb500 sp=0xc0000eb4b8 pc=0x5001a5
net.(*TCPConn).Read(0x0?, {0xc00020e99c?, 0xc0000eb568?, 0x46d449?})
	:1 +0x25 fp=0xc0000eb530 sp=0xc0000eb500 pc=0x50bb25
io.ReadAtLeast({0x5d9640, 0xc000206000}, {0xc00020e99c, 0x4, 0x4}, 0x4)
	/home/michael/sdk/go1.23.0/src/io/io.go:335 +0x90 fp=0xc0000eb578 sp=0xc0000eb530 pc=0x4957d0
io.ReadFull(...)
	/home/michael/sdk/go1.23.0/src/io/io.go:354
encoding/binary.Read({0x5d9640, 0xc000206000}, {0x5da8b0, 0x7059a0}, {0x55e7c0, 0xc0000eb6a0})
	/home/michael/sdk/go1.23.0/src/encoding/binary/binary.go:244 +0xa5 fp=0xc0000eb670 sp=0xc0000eb578 pc=0x5102a5
github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).ReadMsg(0xc00020a100)
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:50 +0x48 fp=0xc0000eb6e8 sp=0xc0000eb670 pc=0x514428
github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).Read(0x7ef5869b9a68?, {0xc000280000, 0x40000, 0x4dd4fb?})
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:72 +0x2f fp=0xc0000eb788 sp=0xc0000eb6e8 pc=0x5145af
bufio.(*Reader).Read(0xc0002020c0, {0xc00020e998, 0x4, 0x40ece5?})
	/home/michael/sdk/go1.23.0/src/bufio/bufio.go:241 +0x197 fp=0xc0000eb7c0 sp=0xc0000eb788 pc=0x4d5a57
io.ReadAtLeast({0x5d93e0, 0xc0002020c0}, {0xc00020e998, 0x4, 0x4}, 0x4)
	/home/michael/sdk/go1.23.0/src/io/io.go:335 +0x90 fp=0xc0000eb808 sp=0xc0000eb7c0 pc=0x4957d0
io.ReadFull(...)
	/home/michael/sdk/go1.23.0/src/io/io.go:354
github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32(0xc000208060)
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:163 +0x4a fp=0xc0000eb850 sp=0xc0000eb808 pc=0x51490a
github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1(0xc000202120, 0x5a9b58)
	/home/michael/kr/rsync/internal/receiver/uidlist.go:16 +0x3d fp=0xc0000eb8c0 sp=0xc0000eb850 pc=0x51fc7d
github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList(0xc000202120)
	/home/michael/kr/rsync/internal/receiver/uidlist.go:52 +0x1dd fp=0xc0000eba08 sp=0xc0000eb8c0 pc=0x51ffbd
github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList(0xc000202120)
	/home/michael/kr/rsync/internal/receiver/flist.go:229 +0x378 fp=0xc0000ebb10 sp=0xc0000eba08 pc=0x51c5b8
github.com/gokrazy/rsync/internal/receivermaincmd.clientRun({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0x7ef53d47efc8, 0xc000206000}, {0x7ffc5866600e, ...}, ...)
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341 +0x5cd fp=0xc0000ebc10 sp=0xc0000ebb10 pc=0x550c2d
github.com/gokrazy/rsync/internal/receivermaincmd.socketClient({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0x7ffc58665ff4?, 0x1?}, {0x7ffc5866600e, ...})
	/home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44 +0x425 fp=0xc0000ebcd0 sp=0xc0000ebc10 pc=0x54c205
github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0xc00007e440, 0x1, 0x2}, ...)
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160 +0x5d7 fp=0xc0000ebdf0 sp=0xc0000ebcd0 pc=0x54f697
github.com/gokrazy/rsync/internal/receivermaincmd.Main({0xc0000160a0, 0x5, 0x5}, {0x5d9280?, 0xc000078058?}, {0x5d92a0?, 0xc000078060?}, {0x5d92a0?, 0xc000078068?})
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394 +0x272 fp=0xc0000ebee8 sp=0xc0000ebdf0 pc=0x5510d2
main.main()
	/home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12 +0x4e fp=0xc0000ebf50 sp=0xc0000ebee8 pc=0x5515ae
runtime.main()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:272 +0x28b fp=0xc0000ebfe0 sp=0xc0000ebf50 pc=0x438d4b
runtime.goexit({})
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000ebfe8 sp=0xc0000ebfe0 pc=0x472e61

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:424 +0xce fp=0xc000074fa8 sp=0xc000074f88 pc=0x46bc0e
runtime.goparkunlock(...)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:430
runtime.forcegchelper()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:337 +0xb3 fp=0xc000074fe0 sp=0xc000074fa8 pc=0x439093
runtime.goexit({})
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x472e61
created by runtime.init.7 in goroutine 1
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:325 +0x1a

Phew! This output is pretty dense.

We can use the https://github.com/maruel/panicparse program to present this stack trace in a more colorful and much shorter version:

The functions helpfully highlighted in red are where the problem lies: My rsync receiver implementation was incorrectly expecting the server to send a uid/gid list, despite the PreserveUid and PreserveGid options not being enabled. Commit 6c89d4d fixes the issue.

Tip 2: Attach the delve debugger to the process

If dumping the stack trace in the moment is not sufficient to diagnose the problem, you can go one step further and reach for an interactive debugger.

The most well-known Linux debugger is probably GDB, but when working with Go, I recommend using the delve debugger instead as it typically works better. Install delve if you haven’t already:

% go install github.com/go-delve/delve/cmd/dlv@latest

In this article, I am using delve v1.24.0.

Note: If you want to explore local variables, you should rebuild your program without optimizations and inlining (see the dlv exec docs):

% go install -gcflags=all="-N -l" ./cmd/...

While you can run a new child process in a debugger (use dlv exec) without any special permissions, attaching existing processes in a debugger is disabled by default in Linux for security reasons. We can allow this feature (remember to turn it off later!) using:

% sudo sysctl -w kernel.yama.ptrace_scope=0
kernel.yama.ptrace_scope = 0

…and then we can just dlv attach to the hanging gokr-rsync process:

% dlv attach $(pidof gokr-rsync)
Type 'help' for list of commands.
(dlv)

Great. But if we just print a stack trace, we only see functions from the runtime package:

(dlv) bt
0  0x000000000047bb83 in runtime.futex
   at /home/michael/sdk/go1.23.6/src/runtime/sys_linux_amd64.s:558
1  0x00000000004374d0 in runtime.futexsleep
   at /home/michael/sdk/go1.23.6/src/runtime/os_linux.go:69
2  0x000000000040d89d in runtime.notesleep
   at /home/michael/sdk/go1.23.6/src/runtime/lock_futex.go:170
3  0x000000000044123e in runtime.mPark
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:1866
4  0x000000000044290d in runtime.stopm
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:2886
5  0x00000000004433d0 in runtime.findRunnable
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:3623
6  0x0000000000444e1d in runtime.schedule
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:3996
7  0x00000000004451cb in runtime.park_m
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:4103
8  0x0000000000477eee in runtime.mcall
   at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:459

The reason is that no goroutine is running (the program is waiting indefinitely to receive data from the server), so we see one of the OS threads waiting in the Go scheduler.

We first need to switch to the goroutine we are interested in (grs prints all goroutines), and then the stack trace looks like what we expect:

(dlv) gr 1
Switched from 0 to 1 (thread 414327)
(dlv) bt
 0  0x0000000000474ebc in runtime.gopark
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:425
 1  0x000000000043819e in runtime.netpollblock
    at /home/michael/sdk/go1.23.6/src/runtime/netpoll.go:575
 2  0x000000000047435c in internal/poll.runtime_pollWait
    at /home/michael/sdk/go1.23.6/src/runtime/netpoll.go:351
 3  0x00000000004ed15a in internal/poll.(*pollDesc).wait
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_poll_runtime.go:84
 4  0x00000000004ed1f1 in internal/poll.(*pollDesc).waitRead
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_poll_runtime.go:89
 5  0x00000000004ee351 in internal/poll.(*FD).Read
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_unix.go:165
 6  0x0000000000569bb3 in net.(*netFD).Read
    at /home/michael/sdk/go1.23.6/src/net/fd_posix.go:55
 7  0x000000000057a025 in net.(*conn).Read
    at /home/michael/sdk/go1.23.6/src/net/net.go:189
 8  0x000000000058fcc5 in net.(*TCPConn).Read
    at :1
 9  0x00000000004b72e8 in io.ReadAtLeast
    at /home/michael/sdk/go1.23.6/src/io/io.go:335
10  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
11  0x0000000000598d5f in encoding/binary.Read
    at /home/michael/sdk/go1.23.6/src/encoding/binary/binary.go:244
12  0x00000000005a0b7a in github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).ReadMsg
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:50
13  0x00000000005a0f17 in github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).Read
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:72
14  0x0000000000528de8 in bufio.(*Reader).Read
    at /home/michael/sdk/go1.23.6/src/bufio/bufio.go:241
15  0x00000000004b72e8 in io.ReadAtLeast
    at /home/michael/sdk/go1.23.6/src/io/io.go:335
16  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
17  0x00000000005a19ef in github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:163
18  0x00000000005b77d2 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:16
19  0x00000000005b7ea8 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:52
20  0x00000000005b18db in github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList
    at /home/michael/kr/rsync/internal/receiver/flist.go:229
21  0x0000000000605390 in github.com/gokrazy/rsync/internal/receivermaincmd.clientRun
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341
22  0x00000000005fe572 in github.com/gokrazy/rsync/internal/receivermaincmd.socketClient
    at /home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44
23  0x0000000000602f10 in github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160
24  0x0000000000605e7e in github.com/gokrazy/rsync/internal/receivermaincmd.Main
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394
25  0x0000000000606653 in main.main
    at /home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12
26  0x000000000043fa47 in runtime.main
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:272
27  0x000000000047bd01 in runtime.goexit
    at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:1700

Tip 3: Save a core dump for later

If you don’t have time to poke around in the debugger now, you can save a core dump for later.

In addition to printing the stack trace on SIGQUIT, we can make the Go runtime crash the program, which in turn makes the Linux kernel write a core dump, by running our program with the environment variable GOTRACEBACK=crash.

Modern Linux systems typically include systemd-coredump(8) (but you might need to explicitly install it, for example on Ubuntu) to collect core dumps (and remove old ones). You can use coredumpctl(1) to list and work with them. On macOS, collecting cores is more involved. I don’t know about Windows.

In case your Linux system does not use systemd-coredump, you can use ulimit -c unlimited and set the kernel’s kernel.core_pattern sysctl setting. You can find more details and options in the CoreDumpDebugging page of the Go wiki. For this article, we will stick to coredumpctl:

% GOTRACEBACK=crash gokr-rsync -rtO --delete rsync://rsync.paas.rpki.ripe.net/repo/ /tmp/rpki-repo
[…]
^\SIGQUIT: quit
[…]
zsh: IOT instruction (core dumped)  GOTRACEBACK=crash gokr-rsync -rtO […]

The last line is what we want to see: it should say “core dumped”.

This core should now show up in coredumpctl(1) :

% coredumpctl info
           PID: 414607 (gokr-rsync)
           UID: 1000 (michael)
           GID: 1000 (michael)
        Signal: 6 (ABRT)
     Timestamp: Sat 2025-02-08 10:18:27 CET (12s ago)
  Command Line: gokr-rsync -rtO --delete rsync://rsync.paas.rpki.ripe.net/repo/ /tmp/rpki-repo
    Executable: /bin/gokr-rsync
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (michael)
       Boot ID: 6158dd3b52af4b8384c103a8a336fc02
    Machine ID: ecb5a44f1a5846ad871566e113bf8937
      Hostname: midna
       Storage: /var/lib/systemd/coredump/core.gokr-rsync.1000.6158dd3b52af4b8384c103a8a336fc02.414607.1739006307000000.zst (present)
  Size on Disk: 158.3K
       Message: Process 414607 (gokr-rsync) of user 1000 dumped core.
                
    Module [dso] without build-id.
    Module [dso]
    Stack trace of thread 1604447:
    #0  0x0000000000475a41 runtime.raise.abi0 (/bin/gokr-rsync + 0x75a41)
    #1  0x0000000000451d85 runtime.dieFromSignal (/bin/gokr-rsync + 0x51d85)
    #2  0x00000000004522e6 runtime.sigfwdgo (/bin/gokr-rsync + 0x522e6)
    #3  0x0000000000450c45 runtime.sigtrampgo (/bin/gokr-rsync + 0x50c45)
    #4  0x0000000000475d26 runtime.sigtramp.abi0 (/bin/gokr-rsync + 0x75d26)
    #5  0x0000000000475e20 n/a (/bin/gokr-rsync + 0x75e20)
    ELF object binary architecture: AMD x86-64

If you see only hexadecimal addresses followed by n/a (n/a + 0x0), that means systemd-coredump could not symbolize (= resolve addresses to function names) your core dump. Here are a few possible reasons for missing symbolization:

Linux 6.12 and 6.13 produced core dumps that elfutils cannot symbolize. systemd-coredump uses elfutils for symbolization, so avoid 6.12/6.13 in favor of using 6.14 or newer.
With systemd v234-v256, systemd-coredump did not have permission to look into programs living in the /home directory (fixed with commit 4ac1755 in systemd v257+).
- Similarly, systemd-coredump runs with PrivateTmp=yes, meaning it won’t be able to access programs you place in /tmp.
Go builds with debug symbols by default, but maybe you are explicitly stripping debug symbols in your build, by building with -ldflags=-w?

We can now use coredumpctl(1) to launch delve for this program + core dump:

% coredumpctl debug --debugger=dlv --debugger-arguments=core
[…]
Type 'help' for list of commands.
(dlv) gr 1
Switched from 0 to 1 (thread 414607)
(dlv) bt
[…]
16  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
17  0x00000000005a19ef in github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:163
18  0x00000000005b77d2 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:16
19  0x00000000005b7ea8 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:52
20  0x00000000005b18db in github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList
    at /home/michael/kr/rsync/internal/receiver/flist.go:229
21  0x0000000000605390 in github.com/gokrazy/rsync/internal/receivermaincmd.clientRun
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341
22  0x00000000005fe572 in github.com/gokrazy/rsync/internal/receivermaincmd.socketClient
    at /home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44
23  0x0000000000602f10 in github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160
24  0x0000000000605e7e in github.com/gokrazy/rsync/internal/receivermaincmd.Main
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394
25  0x0000000000606653 in main.main
    at /home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12
26  0x000000000043fa47 in runtime.main
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:272
27  0x000000000047bd01 in runtime.goexit
    at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:1700

Conclusion

In my experience, in the medium to long term, it always pays off to set up your environment such that you can debug your programs conveniently. I strongly encourage every programmer (and even users!) to invest time into your development and debugging setup.

Luckily, Go comes with stack printing functionality by default (just press Ctrl+\) and we can easily get a core dump out of our Go programs by running them with GOTRACEBACK=crash — provided the system is set up to collect core dumps.

Together with the delve debugger, this gives us all we need to effectively and efficiently diagnose problems in Go programs.

Go Protobuf: The new Opaque API

2024-12-21T11:06:00+01:00

[Protocol Buffers (Protobuf) is Google’s language-neutral data interchange format. See protobuf.dev.]

Back in March 2020, we released a major overhaul of the Go Protobuf API. The google.golang.org/protobuf package introduced first-class support for reflection, a dynamicpb implementation and the protocmp package for easier testing.

That release introduced a new protobuf module with a new API. Today, we are releasing an additional API for generated code, meaning the Go code in the .pb.go files created by the protocol compiler (protoc). This blog post explains our motivation for creating a new API and shows you how to use it in your projects.

To be clear: We are not removing anything. We will continue to support the existing API for generated code, just like we still support the older protobuf module (by wrapping the google.golang.org/protobuf implementation). Go is committed to backwards compatibility and this applies to Go Protobuf, too!

Background: the (existing) Open Struct API

We now call the existing API the Open Struct API, because generated struct types are open to direct access. In the next section, we will see how it differs from the new Opaque API.

To work with protocol buffers, you first create a .proto definition file like this one:

edition = "2023";  // successor to proto2 and proto3

package log;

message LogEntry {
  string backend_server = 1;
  uint32 request_size = 2;
  string ip_address = 3;
}

Then, you run the protocol compiler (protoc) to generate code like the following (in a .pb.go file):

package logpb

type LogEntry struct {
  BackendServer *string
  RequestSize   *uint32
  IPAddress     *string
  // …internal fields elided…
}

func (l *LogEntry) GetBackendServer() string { … }
func (l *LogEntry) GetRequestSize() uint32   { … }
func (l *LogEntry) GetIPAddress() string     { … }

Now you can import the generated logpb package from your Go code and call functions like proto.Marshal to encode logpb.LogEntry messages into protobuf wire format.

You can find more details in the Generated Code API documentation.

(Existing) Open Struct API: Field Presence

An important aspect of this generated code is how field presence (whether a field is set or not) is modeled. For instance, the above example models presence using pointers, so you could set the BackendServer field to:

proto.String("zrh01.prod"): the field is set and contains “zrh01.prod”
proto.String(""): the field is set (non-nil pointer) but contains an empty value
nil pointer: the field is not set

If you are used to generated code not having pointers, you are probably using .proto files that start with syntax = "proto3". The field presence behavior changed over the years:

syntax = "proto2" uses explicit presence by default
syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default

The new Opaque API

We created the new Opaque API to uncouple the Generated Code API from the underlying in-memory representation. The (existing) Open Struct API has no such separation: it allows programs direct access to the protobuf message memory. For example, one could use the flag package to parse command-line flag values into protobuf message fields:

var req logpb.LogEntry
flag.StringVar(&req.BackendServer, "backend", os.Getenv("HOST"), "…")
flag.Parse() // fills the BackendServer field from -backend flag

The problem with such a tight coupling is that we can never change how we lay out protobuf messages in memory. Lifting this restriction enables many implementation improvements, which we’ll see below.

What changes with the new Opaque API? Here is how the generated code from the above example would change:

package logpb

type LogEntry struct {
  xxx_hidden_BackendServer *string // no longer exported
  xxx_hidden_RequestSize   uint32  // no longer exported
  xxx_hidden_IPAddress     *string // no longer exported
  // …internal fields elided…
}

func (l *LogEntry) GetBackendServer() string { … }
func (l *LogEntry) HasBackendServer() bool   { … }
func (l *LogEntry) SetBackendServer(string)  { … }
func (l *LogEntry) ClearBackendServer()      { … }
// …

With the Opaque API, the struct fields are hidden and can no longer be directly accessed. Instead, the new accessor methods allow for getting, setting, or clearing a field.

Opaque structs use less memory

One change we made to the memory layout is to model field presence for elementary fields more efficiently:

The (existing) Open Struct API uses pointers, which adds a 64-bit word to the space cost of the field.
The Opaque API uses bit fields, which require one bit per field (ignoring padding overhead).

Using fewer variables and pointers also lowers load on the allocator and on the garbage collector.

The performance improvement depends heavily on the shapes of your protocol messages: The change only affects elementary fields like integers, bools, enums, and floats, but not strings, repeated fields, or submessages (because it is less profitable for those types).

Our benchmark results show that messages with few elementary fields exhibit performance that is as good as before, whereas messages with more elementary fields are decoded with significantly fewer allocations:

             │ Open Struct API │             Opaque API             │
             │    allocs/op    │  allocs/op   vs base               │
Prod#1          360.3k ± 0%       360.3k ± 0%  +0.00% (p=0.002 n=6)
Search#1       1413.7k ± 0%       762.3k ± 0%  -46.08% (p=0.002 n=6)
Search#2        314.8k ± 0%       132.4k ± 0%  -57.95% (p=0.002 n=6)

Reducing allocations also makes decoding protobuf messages more efficient:

             │ Open Struct API │             Opaque API            │
             │   user-sec/op   │ user-sec/op  vs base              │
Prod#1         55.55m ± 6%        55.28m ± 4%  ~ (p=0.180 n=6)
Search#1       324.3m ± 22%       292.0m ± 6%  -9.97% (p=0.015 n=6)
Search#2       67.53m ± 10%       45.04m ± 8%  -33.29% (p=0.002 n=6)

(All measurements done on an AMD Castle Peak Zen 2. Results on ARM and Intel CPUs are similar.)

Note: proto3 with implicit presence similarly does not use pointers, so you will not see a performance improvement if you are coming from proto3. If you were using implicit presence for performance reasons, forgoing the convenience of being able to distinguish empty fields from unset ones, then the Opaque API now makes it possible to use explicit presence without a performance penalty.

Motivation: Lazy Decoding

Lazy decoding is a performance optimization where the contents of a submessage are decoded when first accessed instead of during proto.Unmarshal. Lazy decoding can improve performance by avoiding unnecessarily decoding fields which are never accessed.

Lazy decoding can’t be supported safely by the (existing) Open Struct API. While the Open Struct API provides getters, leaving the (un-decoded) struct fields exposed would be extremely error-prone. To ensure that the decoding logic runs immediately before the field is first accessed, we must make the field private and mediate all accesses to it through getter and setter functions.

This approach made it possible to implement lazy decoding with the Opaque API. Of course, not every workload will benefit from this optimization, but for those that do benefit, the results can be spectacular: We have seen logs analysis pipelines that discard messages based on a top-level message condition (e.g. whether backend_server is one of the machines running a new Linux kernel version) and can skip decoding deeply nested subtrees of messages.

As an example, here are the results of the micro-benchmark we included, demonstrating how lazy decoding saves over 50% of the work and over 87% of allocations!

                  │   nolazy    │                lazy                │
                  │   sec/op    │   sec/op     vs base               │
Unmarshal/lazy-24   6.742µ ± 0%   2.816µ ± 0%  -58.23% (p=0.002 n=6)

                  │    nolazy    │                lazy                 │
                  │     B/op     │     B/op      vs base               │
Unmarshal/lazy-24   3.666Ki ± 0%   1.814Ki ± 0%  -50.51% (p=0.002 n=6)

                  │   nolazy    │               lazy                │
                  │  allocs/op  │ allocs/op   vs base               │
Unmarshal/lazy-24   64.000 ± 0%   8.000 ± 0%  -87.50% (p=0.002 n=6)

Motivation: reduce pointer comparison mistakes

Modeling field presence with pointers invites pointer-related bugs.

Consider an enum, declared within the LogEntry message:

message LogEntry {
  enum DeviceType {
    DESKTOP = 0;
    MOBILE = 1;
    VR = 2;
  };
  DeviceType device_type = 1;
}

A simple mistake is to compare the device_type enum field like so:

if cv.DeviceType == logpb.LogEntry_DESKTOP.Enum() { // incorrect!

Did you spot the bug? The condition compares the memory address instead of the value. Because the Enum() accessor allocates a new variable on each call, the condition can never be true. The check should have read:

if cv.GetDeviceType() == logpb.LogEntry_DESKTOP {

The new Opaque API prevents this mistake: Because fields are hidden, all access must go through the getter.

Motivation: reduce accidental sharing mistakes

Let’s consider a slightly more involved pointer-related bug. Assume you are trying to stabilize an RPC service that fails under high load. The following part of the request middleware looks correct, but still the entire service goes down whenever just one customer sends a high volume of requests:

logEntry.IPAddress = req.IPAddress
logEntry.BackendServer = proto.String(hostname)
// The redactIP() function redacts IPAddress to 127.0.0.1,
// unexpectedly not just in logEntry *but also* in req!
go auditlog(redactIP(logEntry))
if quotaExceeded(req) {
	// BUG: All requests end up here, regardless of their source.
	return fmt.Errorf("server overloaded")
}

Did you spot the bug? The first line accidentally copied the pointer (thereby sharing the pointed-to variable between the logEntry and req messages) instead of its value. It should have read:

logEntry.IPAddress = proto.String(req.GetIPAddress())

The new Opaque API prevents this problem as the setter takes a value (string) instead of a pointer:

logEntry.SetIPAddress(req.GetIPAddress())

Motivation: Fix Sharp Edges: reflection

To write code that works not only with a specific message type (e.g. logpb.LogEntry), but with any message type, one needs some kind of reflection. The previous example used a function to redact IP addresses. To work with any type of message, it could have been defined as func redactIP(proto.Message) proto.Message { … }.

Many years ago, your only option to implement a function like redactIP was to reach for Go’s reflect package, which resulted in very tight coupling: you had only the generator output and had to reverse-engineer what the input protobuf message definition might have looked like. The google.golang.org/protobuf module release (from March 2020) introduced Protobuf reflection, which should always be preferred: Go’s reflect package traverses the data structure’s representation, which should be an implementation detail. Protobuf reflection traverses the logical tree of protocol messages without regard to its representation.

Unfortunately, merely providing protobuf reflection is not sufficient and still leaves some sharp edges exposed: In some cases, users might accidentally use Go reflection instead of protobuf reflection.

For example, encoding a protobuf message with the encoding/json package (which uses Go reflection) was technically possible, but the result is not canonical Protobuf JSON encoding. Use the protojson package instead.

The new Opaque API prevents this problem because the message struct fields are hidden: accidental usage of Go reflection will see an empty message. This is clear enough to steer developers towards protobuf reflection.

Motivation: Making the ideal memory layout possible

The benchmark results from the More Efficient Memory Representation section have already shown that protobuf performance heavily depends on the specific usage: How are the messages defined? Which fields are set?

To keep Go Protobuf as fast as possible for everyone, we cannot implement optimizations that help only one program, but hurt the performance of other programs.

The Go compiler used to be in a similar situation, up until Go 1.20 introduced Profile-Guided Optimization (PGO). By recording the production behavior (through profiling) and feeding that profile back to the compiler, we allow the compiler to make better trade-offs for a specific program or workload.

We think using profiles to optimize for specific workloads is a promising approach for further Go Protobuf optimizations. The Opaque API makes those possible: Program code uses accessors and does not need to be updated when the memory representation changes, so we could, for example, move rarely set fields into an overflow struct.

Migration

You can migrate on your own schedule, or even not at all—the (existing) Open Struct API will not be removed. But, if you’re not on the new Opaque API, you won’t benefit from its improved performance, or future optimizations that target it.

We recommend you select the Opaque API for new development. Protobuf Edition 2024 (see Protobuf Editions Overview if you are not yet familiar) will make the Opaque API the default.

The Hybrid API

Aside from the Open Struct API and Opaque API, there is also the Hybrid API, which keeps existing code working by keeping struct fields exported, but also enabling migration to the Opaque API by adding the new accessor methods.

With the Hybrid API, the protobuf compiler will generate code on two API levels: the .pb.go is on the Hybrid API, whereas the _protoopaque.pb.go version is on the Opaque API and can be selected by building with the protoopaque build tag.

Rewriting Code to the Opaque API

See the migration guide for detailed instructions. The high-level steps are:

Enable the Hybrid API.
Update existing code using the open2opaque migration tool.
Switch to the Opaque API.

Advice for published generated code: Use Hybrid API

Small usages of protobuf can live entirely within the same repository, but usually, .proto files are shared between different projects that are owned by different teams. An obvious example is when different companies are involved: To call Google APIs (with protobuf), use the Google Cloud Client Libraries for Go from your project. Switching the Cloud Client Libraries to the Opaque API is not an option, as that would be a breaking API change, but switching to the Hybrid API is safe.

Our advice for such packages that publish generated code (.pb.go files) is to switch to the Hybrid API please! Publish both the .pb.go and the _protoopaque.pb.go files, please. The protoopaque version allows your consumers to migrate on their own schedule.

Enabling Lazy Decoding

Lazy decoding is available (but not enabled) once you migrate to the Opaque API! 🎉

To enable: in your .proto file, annotate your message-typed fields with the [lazy = true] annotation.

To opt out of lazy decoding (despite .proto annotations), the protolazy package documentation describes the available opt-outs, which affect either an individual Unmarshal operation or the entire program.

Next Steps

By using the open2opaque tool in an automated fashion over the last few years, we have converted the vast majority of Google’s .proto files and Go code to the Opaque API. We continuously improved the Opaque API implementation as we moved more and more production workloads to it.

Therefore, we expect you should not encounter problems when trying the Opaque API. In case you do encounter any issues after all, please let us know on the Go Protobuf issue tracker.

Reference documentation for Go Protobuf can be found on protobuf.dev → Go Reference.

Get a solar panel for your balcony now ☀️

2024-12-10T17:13:00+01:00

A year ago, I got a solar panel for my balcony — an easy way to vote with your wallet to convert more of the world’s energy usage to solar power. That was a great decision and I would recommend everyone get a solar panel (or two)!

It’s called plug-in solar panel because you can just plug it in

In my experience, many people are surprised about the basics of how power works: You do not need to connect devices to a battery in order to enjoy solar power. You can just plug in the solar panel into your household electricity setup. Any of your consumers (like a TV, or electric cooktop) will now use the power that your solar panel produces before consuming power from the grid.

Here’s the panel I have (Weber barbecue for scale). As you can see, the panel is not yet mounted at an angle, just hung over the balcony. The black box at the back of the panel is the inverter (“Wechselrichter”). You connect the panel on one side and get electricity out the other side.

Which solar panel to buy?

There are two big questions to answer when chosing a solar panel: what peak capacity should your panel(s) have and which company / seller do you buy from?

Regarding panel capacity: When I look at my energy usage, I see about 100 watts of baseline load. This includes always-on servers and other home automation devices. During working hours, running a PC and (power-hungry) monitor adds another 100 watts or so. Around noon, there is quite a spike in usage when cooking with my induction cooktop.

Hence, I figured a plug & play solar panel with the smallest size of 385 Wp would be well equipped to cover baseline usage, compared to the next bigger unit with 780 Wp, which seems oversized for my usage. Note that a peak capacity of 385 Wp will not necessarily mean that you will measure 380W of output. I did repeatedly measure energy production exceeding 300W.

Regarding the company, the best offer I found in Switzerland was a small company called erneuer.bar, which means “renewable” in German. They ship the panels with barely any packaging in fully electric vehicles and their offer is eligible for the topten bonus program from EWZ, meaning you’ll get back 200 CHF if you fill in a form.

The specific model I ordered was called “385 Wp Plug & Play Solar (DE)”. Here’s the bill:

Produkt	Preis
385 Wp Plug & Play Solar (DE)	CHF 520.00
Mounting kit: balcony, 1 panel	CHF 75.00
Pre-mount: mounting kit balcony	CHF 60.00
WiFi measurement myStrom	CHF 55.00
Shipping	CHF 68.00
Total	CHF 778.00

Of course, you can save some money in various ways. For example, the measurement device and pre-mount option are both not required, but convenient. Similarly, you can probably find solar panels for cheaper, but the offer that erneuer.bar has put together truly is very convenient and works well, and to me that’s worth some money.

One mistake I made when ordering is selecting a 5m cable. It turned out I needed a 10m cable, so I recommend you measure better than I did (or just select the longer cable). On the plus side, customer service was excellent: I quickly received an email response and could just send back my cable in exchange for a new one.

Amortization? Who cares!

Many people seem to consider only the financial aspect of buying a solar panel and calculate when the solar panel will have paid for itself. I don’t care. My goal is to convert more energy usage to green energy, not to save money.

Similarly, some people install batteries so that they can use “their” energy for themselves, in case the solar panel produces more than they use at that moment. I couldn’t care less who uses the energy I produce — as long as it’s green energy, anyone is welcome to consume it.

(Of course I understand these questions become more important the larger a solar installation gets. But we’re talking about one balcony and one solar panel (or two) covering someone’s baseline residential household electricity load. Don’t overthink it!)

Requirement: balcony power socket

Aside from having a balcony, there is only one hard requirement: you need a power socket.

This requirement is either trivially fulfilled if you already have an outdoor power socket on your balcony (lucky you!), or might turn out to be the most involved part of the project. Either way, because an electrician needs to install power sockets, all you can do is get permission from your landlord and make an appointment with your electrician of choice.

In terms of cost, you will probably spend a few hundred bucks, depending on your area’s cost of living. A good idea that did not occur to me back then: Ask around in your house if any neighbors would be interested in getting a balcony power socket, too, and do it all in one go (for cheaper).

Bureaucracy

One can easily find stories online about electricity providers and landlords not permitting the installation of solar panels for… rather questionable reasons. For example, some claimed that solar panels could overload the house electricity infrastructure! A drastic-sounding claim, but nonsense in practice. Luckily, law makers are recognizing this and are removing barriers.

Electricity provider and the law

In Switzerland 🇨🇭, you can connect panels producing up to 600W without an electrician, but you need to notify your electricity provider.

In Germany 🇩🇪, you can connect panels producing up to 800W (as of May 16th 2024) without an electrician, but you need to register with the Bundesnetzagentur.

Be sure to check your country’s laws and your electricity provider’s rules and processes.

Landlord and neighbors

In Switzerland 🇨🇭, you need to ask your landlord for permission because if your solar panel were to fall down from the balcony, the landlord would be liable. Usually, the landlord insists on proper mounting and the tenant taking over liability. In my case, the landlord also asked me to ensure the neighbors wouldn’t mind. I put up a letter, nobody complained, the landlord accepted.

In Germany 🇩🇪, you do need to ask your landlord for permission, but the landlord pretty much has to agree (as of October 17th 2024). The question is not “if”, but “how” the landlord wants you to install the solar panel.

Optimizing the installation angle

Earlier I wrote that you can just hang the solar panel onto your balcony and plug it in. While this is true, there is one factor that is worth optimizing (as time permits): the installation angle.

If you want more details about the physics background and various considerations that go into chasing the optimal angle, check out these (German) articles about optimizing the installation angle (at Golem) or sizing solar installations (at Heise). I’ll summarize: the angle is important and can result in twice as much energy production! Any angle is usually better than no angle.

In my case, I first “installed” the solar panel (no angle) at 2023-09-30.

Then, about a month later, I installed it at an angle at 2023-10-28.

I unfortunately don’t have a great before/after graph because after I installed the proper angle mount, there were almost no sunny days.

Instead, I will show you data from a comparable time range (early October) in 2023 (before mounting the panel at an angle) and in 2024 (with a properly mounted panel). As you can see, the difference is not that huge, but clearly visible: without an angle mount, I could never exceed 300 Wh per day. With a proper mount, a number of days exceed 300 Wh:

	1st	2nd	3rd	4th	5th	6th	7th	8th	9th	10th
2023 🌞 Wh	133	268	262	208	271	255	274	277	275	194
2024 🌞 Wh	529	119	246	205	160	324	265	335	73	444

How much electricity does my panel generate?

The exact electricity production numbers depend on how much sun ends up on the solar panel. This in turn depends on the weather and how obstructed the solar panel is (neighbors, trees, …).

I like measuring things, so I will share some measurements to give you a rough idea. But note that measuring your solar panel is strictly optional.

On the best recorded day, my panel produced about 1.680 kWh of energy:

The missing parts before 14:00 are caused by the neighbor’s house blocking the sun.

Now, compare this best case with the worst case, a January day with little sun (< 50 Wh):

Let’s zoom out a bit and consider an entire year instead.

In 2024, the panel produced over 177 kWh so far, or, averaged to the daily value, ≈0.5 kWh/day:

Or, in numeric form (all numbers in kWh):

Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2.9	6.6	10.9	18.4	29.1	27.7	37.6	22.1	12.0	6.5	3.3	n/a

Conclusion

A solar panel is a great project to make incremental progress on. It’s just 3 to 4 simple steps, each of which is valuable on its own:

Check with your landlord that installing an outdoor power socket and solar panel is okay.
- Even if you personally do not go any further with your project, you can share the result with your neighbors, who might…
Order an outdoor power socket from your (or your landlord’s) preferred electrician.
- Power will come in handy for lighting when spending summer evenings on the balcony.
Order a solar panel and plug it in.
Optional, but recommended: Optimize the mounting angle later.

That’s it! Come on, get started right away 🌞