Michael Stapelbergs Website: posts tagged nix

Coding Agent VMs on NixOS with microvm.nix

2026-02-01T09:00:00+01:00

I have come to appreciate coding agents to be valuable tools for working with computer program code in any capacity, such as learning about any program’s architecture, diagnosing bugs or developing proofs of concept. Depending on the use-case, reviewing each command the agent wants to run can get tedious and time-consuming very quickly. To safely run a coding agent without review, I wanted a Virtual Machine (VM) solution where the agent has no access to my personal files and where it’s no big deal if the agent gets compromised by malware: I can just throw away the VM and start over.

Instead of setting up a stateful VM and re-installing it when needed (ugh!), I prefer the model of ephemeral VMs where nothing persists on disk, except for what is explicitly shared with the host.

The microvm.nix project makes it easy to create such VMs on NixOS, and this article shows you how I like to set up my VMs.

Setting up microvm.nix

Let’s jump right in! The next sections walk you through how I set up my config.

Step 1: network prep

First, I created a new microbr bridge which uses 192.168.33.1/24 as IP address range and NATs out of the eno1 network interface. All microvm* interfaces will be added to that bridge:

systemd.network.netdevs."20-microbr".netdevConfig = {
  Kind = "bridge";
  Name = "microbr";
};

systemd.network.networks."20-microbr" = {
  matchConfig.Name = "microbr";
  addresses = [ { Address = "192.168.83.1/24"; } ];
  networkConfig = {
    ConfigureWithoutCarrier = true;
  };
};

systemd.network.networks."21-microvm-tap" = {
  matchConfig.Name = "microvm*";
  networkConfig.Bridge = "microbr";
};

networking.nat = {
  enable = true;
  internalInterfaces = [ "microbr" ];
  externalInterface = "eno1";
};

Step 2: `flake.nix`

Then, I added the microvm module as a new input to my flake.nix (check out the microvm.nix documentation for details) and enabled the microvm.nixosModules.host module on the NixOS configuration for my PC (midna). I also created a new microvm.nix file, in which I declare all my VMs. Here’s what my flake.nix looks like:

{
  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-25.11";
    };
    # For more recent claude-code
    nixpkgs-unstable = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    stapelbergnix = {
      url = "github:stapelberg/nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    zkjnastools = {
      url = "github:stapelberg/zkj-nas-tools";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    microvm = {
      url = "github:microvm-nix/microvm.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    home-manager = {
      url = "github:nix-community/home-manager/release-25.11";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    configfiles = {
      url = "github:stapelberg/configfiles";
      flake = false; # repo is not a flake
    };
  };

  outputs =
    {
      self,
      stapelbergnix,
      zkjnastools,
      nixpkgs,
      nixpkgs-unstable,
      microvm,
      home-manager,
      configfiles,
    }@inputs:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
      pkgs-unstable = import nixpkgs-unstable {
        inherit system;
        config.allowUnfree = true;
      };
    in
    {
      nixosConfigurations = {
        midna = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          specialArgs = { inherit inputs; };
          modules = [
            (import ./configuration.nix)
            stapelbergnix.lib.userSettings
            # Use systemd for network configuration
            stapelbergnix.lib.systemdNetwork
            # Use systemd-boot as bootloader
            stapelbergnix.lib.systemdBoot
            # Run prometheus node exporter in tailnet
            stapelbergnix.lib.prometheusNode
            zkjnastools.nixosModules.zkjbackup
            microvm.nixosModules.host
            ./microvm.nix
          ];
        };
      };
    };
}

Step 3: `microvm.nix`

The following microvm.nix declares two microvms, one for Emacs (about which I wanted to learn more) and one for Go Protobuf, a code base I am familiar with and can use to understand Claude’s capabilities:

{
  config,
  lib,
  pkgs,
  inputs,
  ...
}:

let
  inherit (inputs)
    nixpkgs-unstable
    stapelbergnix
    microvm
    configfiles
    home-manager
    ;

  microvmBase = import ./microvm-base.nix;
in
{
  microvm.vms.emacsvm = {
    autostart = false;
    config = {
      imports = [
        stapelbergnix.lib.userSettings
        microvm.nixosModules.microvm
        (microvmBase {
          hostName = "emacsvm";
          ipAddress = "192.168.83.6";
          tapId = "microvm4";
          mac = "02:00:00:00:00:05";
          workspace = "/home/michael/microvm/emacs";
          inherit
            nixpkgs-unstable
            configfiles
            home-manager
            stapelbergnix
            ;
        })
        ./microvms/emacs.nix
      ];
    };
  };

  microvm.vms.goprotobufvm = {
    autostart = false;
    config = {
      imports = [
        stapelbergnix.lib.userSettings
        microvm.nixosModules.microvm
        (microvmBase {
          hostName = "goprotobufvm";
          ipAddress = "192.168.83.7";
          tapId = "microvm5";
          mac = "02:00:00:00:00:06";
          workspace = "/home/michael/microvm/goprotobuf";
          inherit
            nixpkgs-unstable
            configfiles
            home-manager
            stapelbergnix
            ;
          extraZshInit = ''
            export GOPATH=$HOME/go
            export PATH=$GOPATH/bin:$PATH
          '';
        })
        ./microvms/goprotobuf.nix
      ];
    };
  };
}

Step 4: `microvm-base.nix`

The microvm-base.nix module takes these parameters and declares:

Network settings: I like using systemd-networkd(8) and systemd-resolved(8) .
Shared directories for:
- the workspace directory, e.g. ~/microvm/emacs
- the host’s Nix store, so the VM can access software from cache (often)
- this VM’s SSH host keys
- ~/claude-microvm, which is a separate state directory, used only on the microvms.
an 8 GB disk overlay (var.img), stored in /var/lib/microvms/
cloud-hypervisor (QEMU also works well!) as the hypervisor, with 8 vCPUs and 4 GB RAM.
A workaround for systemd trying to unmount /nix/store (which causes a deadlock).

Expand full microvm-base.nix code

NixCon 2025 Trip Report 🐝

2025-09-21T09:34:00+02:00

I liked the NixOS meetup earlier this year, and at the end of the meetup they told everyone about NixCon 2025, which would be happening in Switzerland this year, at the very same location, the University Of Applied Sciences OST in Rapperswil, so I decided to go! In this trip report, I want to give you a rough impression of how I experienced this awesome conference :)

The bee in the title is a NixCon inside joke ;)

Friday

I arrived at about 09:30 on a rainy Friday morning, meaning I hurried from the train station into OST building 1 to show my ticket QR code and pick up my conference badge and custom name tag that I pre-ordered. The custom ones have your name engraved and come with a strong magnet to attach them to your clothes:

After grabbing a bite to eat, I headed to the main lecture hall for the opening session. Prof. Dr. Farhad Mehta from OST, as well as the entire NixCon orga team, welcomed the 450 registered attendees to the 10th NixCon! I recognized many familiar faces from the Nix meetup, but many hands went up when the audience was asked for whom it was the first time at NixCon, or in Switzerland in general.

I want to thank Prof. Mehta in particular for making possible such meetups and events! 👏

If you work at a university, school or other organisation that has access to rooms, consider offering to host a meetup (on a regular basis, or even just once)! Locations are always hard to find, so offering a space is a great contribution to Open Source.

“What if GitHub Actions were local-first and built using Nix?”

The first technical talk of the day was “What if GitHub Actions were local-first and built using Nix?” by Domen Kožar, the person behind cachix.org, which is a hosted Nix cache. The talk pitched cloud.devenv.sh, which is a Nix-based CI solution (like GitHub Actions) using devenv.

By using this solution, you solve the problem that you can’t easily / completely run GitHub Actions locally (yes, we all know about act), and you get to (?) write Nix configs instead of YAML configs.

The solution seems nice, but I found the talk a little unstructured because the presenter jumped around between slides so much. One crucial question was left unanswered: How do you integrate this custom solution with your GitHub projects? To me, diverging from the default way of configuring GitHub Actions does not seem worth it for my projects. YMMV.

→ watch the recording (46 minutes) on media.ccc.de

“Rewriting the Hydra Queue Runner in Rust”

Next up: “Rewriting the Hydra Queue Runner in Rust” by Simon Hauser from Helsinki Systems, a small German software company. Hydra is the component in the NixOS infrastructure which schedules builds: when nixpkgs changes, this is the component that runs the build whose result ends up on cache.nixos.org (the Debian equivalent is buildd).

Simon explained that bottlenecks in the current queue runner result in stranding of infrastructure: the project has machines available that it cannot use fully. He outlined how they replaced a crufty SSH-based automation with a well-designed gRPC protocol. I got the impression that a group of people was involved in developing and reviewing this design, which is a great sign for a healthy project.

One thing that was unfortunately missing from the talk were metrics. It would have been great to see a few graphs that illustrate just how much better the rewritten queue runner is.

Currently, the new queue runner is already used for Nix Community builds, but not yet in production for NixOS itself. Hopefully soon, though!

→ watch the recording (27 minutes) on media.ccc.de

“You can’t spell “devshell” without “hell””

This talk was presented by Zach Mitchell from Flox, which is a Nix-based dev environment solution. Thus far, I use nix-shell or nix develop (see Development shells with Nix: four quick examples), so I was curious what I’d learn from this talk.

Zach explained that both, nix-shell and nix develop were originally written to debug Nix package builds, not to provide general-purpose development environments. For users, this manifests as not being able to use your favorite shell — nix develop only supports Bash. One might read about nix develop -c exec , but that’s wrong, because then the shell’s RC files run after Nix setup, possibly destroying parts of the setup.

One interesting thing I learnt is that the Nix garbage collector scans /proc to avoid removing Nix store paths that are still needed by running processes.

Zach mentioned https://github.com/zmitchell/proctrace, which is a bpftrace-based profiler that tracks forks/execs and generates gantt chart syntax of the timing. Sounds cool, but is unfortunately broken right now…? Too bad.

→ watch the recording (45 minutes) on media.ccc.de

“The Nix Binary Cache and AWS”

In this fireside chat, Tarus Balog shared how he ended up at AWS after 20 years of Open Source, and how his team wants to give back to the community. One specific way in which they’re doing that is by hosting cache.nixos.org.

→ watch the recording (24 minutes) on media.ccc.de

“Nix-based development environments at Shopify (reprise)”

Josh Heinrichs from Shopify shared how they adopted Nix (again!), and I think real-world enterprise adoption stories like these are very interesting.

In summary, Shopify had a dev command (since 2016), which offered declarative configuration and then dispatched to apt (Linux) or homebrew (macOS). In the first attempt to move to Nix, the effort didn’t reach stable footing (some folks couldn’t use it yet) and then a company-wide shift to cloud development happened, where the easier solution was to “just use ubuntu”.

A few years in, folks are apparently not so happy with the cloud development environments and one day, Shopify CEO Tobias Lütke finds devenv, which is a Nix-based solution that is remarkably similar to Shopify’s dev. So Tobi adopts devenv for one of their services and becomes supportive of using Nix. This time around, they spend a lot more time on a successful rollout within the organization, meaning incremental adoption, getting all stakeholders on board, etc.

The takeaway is that one specific, well-supported use-case can be the adoption driver. And once you have your development environments on a Nix-based solution, you can more easily adopt other parts of the ecosystem as well.

→ watch the recording (19 minutes) on media.ccc.de

“My first Nix Aha!: A Newcomer’s Perspective”

In a similar spirit to the Shopify talk, Kavisha Kumar from ASML shared how she got into Nix after seeing a colleague use nix-shell to obtain a clean development shell.

Kavisha spent a lot of time at ASML to teach others about why and how to use Nix. She shared a number of nice metaphors that explained Nix concepts through the subject area of video gaming.

I think many people are excited about Nix, but have trouble conveying that excitement to others. Kavisha showed us a good way that worked for her.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day was filled with lightning talks.

Cole Mickens from Determinate Systems explained what features they are currently shipping in their downstream distribution “Determinate Nix” (features will be upstreamed): lazy trees (a performance optimization for evaluating Flakes), parallel evaluation (brings evaluation times down from 16s to 7s) and a native Linux builder for mac. Next up are Flake Schemas, which I haven’t read about yet.

Yvan Sraka from Numtide, a Nix and DevOps consultancy, showed how he manages Linux machines for friends and family with NixOS. He has his own configuration layer on top of NixOS and only uses the system as a base. Most actual programs are used through AppImage, Flatpaks, envfs and nix-ld. The latter two are solutions to use FHS based programs (those that expect /usr/bin and other standard locations to be present) on non-FHS systems like NixOS. I had heard of nix-ld before, but not of envfs.

Jacek Galowicz from Nixcademy showed how to use systemd-sysupdate and systemd-repart to implement A/B style updates with NixOS and systemd. It’s great to see that this technique is more and more mainstream, as I am also using A/B style updating successfully in gokrazy.

Saturday

The weather on Saturday was a lot better, so I made sure to get a seat with a view of Lake Zürich:

“The bikes have been shed: The official Nix formatter”

In this talk, Silvan Mosberger from Tweag (and one of the main NixCon organizers!), explains how the official formatting tool for .nix files came to be.

I was delighted to hear gofmt, the official Go formatter, being mentioned as a source of inspiration. Just like in other language ecosystems, introducing uniform formatting eliminates time-consuming back-and-forth in code review over adhering to coding style. Unfortunately, the formatting folks did not replicate one key aspect to gofmt’s success: gofmt has no options. As the famous Go proverb goes:

Gofmt’s style is no one’s favorite, yet gofmt is everyone’s favorite!

Meaning that it’s more important that everyone uses the same style, compared to everyone being able to express their personal style preferences.

→ watch the recording (20 minutes) on media.ccc.de

“Mastering NixOS Integration Tests: Advanced Techniques for Fast and Robust Multi-VM Tests”

In this two-hour workshop, Jacek Galowicz from Nixcademy, who is not only a Nix teacher, but also happens to be the maintainer of the NixOS integration test driver, shows us how to write complex integration tests with a few lines of Nix and Python.

Jacek showed an integration test example: a Bittorrent service, consisting of tracker, clients, firewalls and multiple networks! Nixpkgs contains over 1000 such integration tests, and running one on your laptop is easy.

The various ways to debug your tests seem pretty cool: using vsock instead of port forwardings, and enabling a debug hook that will make a failed test hang and wait to be debugged.

I thought this was a great overview and Jacek is an engaging teacher. I would recommend booking his classes!

“When Not to Nix: Working with External Config and SOPS Nix”

Ryota spoke about when to use Nix and when not to use Nix. For example, you could manage your dotfiles (config files) with Nix, or you could decide not to. Having recently migrated more and more machines and configurations to Nix, I found myself agreeing with this talk: It’s important to understand what you’ll get out of declaratively or statefully managed configs, and when which approach is better.

→ watch the recording (19 minutes) on media.ccc.de

Lightning Talks

The rest of the day I spent in lightning talks, some of which were sponsored talk slots. I learnt about, in no particular order:

Cloud Hypervisor, a KVM based hypervisor like qemu, but written in Rust.
nixbuild.net, a pay-as-you-go offering for extra build capacity you can rent. On Sunday I heard someone say that their company is using nixbuild.net and it’s very smooth.
NixCI, a Nix-based hosted CI. So, the cloud.devenv.sh service we heard about on Friday is a competitor to this service.
Nix in the Wild is an effort by Flox where they do 45-60 minute interviews about Nix success stories. This might help you convince folks in your organization.
clan is a fleet management solution.
NovaCustom, a one-person laptop/PC company. The laptops come with coreboot and work with NixOS.
ExpressVPN is migrating their internal server setup (TrustedServer) from Debian to NixOS! Deploying weekly in 105+ countries.
Cyberus, a German company, is offering NixOS LTS releases, compliant with the EU Cyber Resilience Act obligations.
David’s styx project is a more bandwidth-efficient download mechanism for NixOS updates. This uses EROFS, which seems like an interesting alternative to SquashFS images.

After all the talks, we met outside for a group picture followed by barbecue at the lake:

NixCon 2025 by Arik Grahl. Licensed under CC BY-SA 4.0.

Sunday

Before the conference, I wasn’t sure if I would even bother showing up for Sunday (Hack day), but on Sunday, I was like “of course!”, and it was a great decision!

Many people were still around and were working on their projects. It felt like the answer to any Nix question was just one chat message away — there was expertise and helping hands from many parts of the project.

I ended up meeting a couple of people I only knew from online interactions before, and we also talked a lot about meetups. Now, I am invited to multiple meetups to give a talk :D

Conclusion

This was a wonderful conference! The orga team and all contributors did a great job!

As always, the OST in Rapperswil is a great venue for Open Source events.

Ticket sales and talk submission / scheduling were done using the Pretix and Pretalx Open Source systems, which makes me proud to have contributed to Pretix.

The selection of talks was great: Some deeply technical, some covering only the human side of things, and many somewhere in between. I got the impression that all the presenters I saw genuinely cared about their topic, so the overall energy was very good!

(You can watch the talk recordings at media.ccc.de: NixCon 2025.)

Also outside of the talks, I had many friendly interactions and interesting conversations. There is a lot of interest and adoption of Nix, which is great to see!

The production level of the conference was very high for such a volunteer-driven event. For example, the very cool sounding break music between talks was created specifically for NixCon: “Lava” by tonstr.studio. Similarly, the welcome bag contained dark Swiss chocolate, specifically made for NixCon (see picture below). I don’t even like dark chocolate, but this one was delicious!

Thanks again to all helpers, and I look forward to coming back soon!

Secret Management on NixOS with sops-nix

2025-08-24T09:56:00+02:00

Passwords and secrets like cryptographic key files are everywhere in computing. When configuring a Linux system, sooner or later you will need to put a password somewhere — for example, when I migrated my existing Linux Network Storage (NAS) setup to NixOS, I needed to specify the desired Samba passwords in my NixOS config (or manage them manually, outside of NixOS). For personal computers, this is fine, but if the goal is to share system configurations (for example in a Git repository), we need a different solution: Secret Management.

What is Secret Management?

The basic idea behind Secret Management systems is to encrypt the secrets at rest, meaning if somebody clones the git repository containing your NixOS system configurations, they cannot access (and therefore, also not deploy) the encrypted secrets.

Conceptually, we need to:

Encrypt the secrets such that the target system can decrypt them.
Encrypt the secrets such that other people working on this config can decrypt them.
Have the target system decrypt secrets at runtime.
Tell our software where to access the decrypted secrets.

sops-nix setup

In this article, I will show how to accomplish the above using sops-nix. Here’s a quick overview of the three different building blocks we will use:

sops is a tool to version-control secrets in git, in their encrypted form.
- sops makes it easy to re-encrypt these secrets when adding/removing authorized keys.
- sops is very flexible and can work with tons of other tools/providers.
sops-nix provides a way to integrate sops with Nix/NixOS
Using sops with age(1) allows us to use our existing SSH private key (humans) or SSH host private key (machines) instead of managing a separate set of key files.

You might wonder why I chose sops-nix over agenix, the other contender? The instructions for setting up sops-nix made more sense to me when I first looked at it, and I wanted to have the option to use sops in other ways, not just with age. If you’re curious about agenix, check out Andreas Gohr’s blog post about agenix.

Step 1. Preparation

I ran the following instructions on an Arch Linux machine on which I installed the Nix tool and enabled Nix Flakes. Follow the link for instructions also for other systems like Debian or Fedora.

Step 2. Obtain an age identity from your personal SSH key

I don’t want to manage an extra key file, so I’ll use ssh-to-age to derive a key from my SSH private key file, which I already take good care of to back up:

midna % mkdir -p $HOME/.config/sops/age/
midna % read -s SSH_TO_AGE_PASSPHRASE; export SSH_TO_AGE_PASSPHRASE
midna % nix run nixpkgs#ssh-to-age -- \
  -private-key \
  -i $HOME/.ssh/id_ed25519 \
  -o $HOME/.config/sops/age/keys.txt

(The SSH_TO_AGE_PASSPHRASE option is documented in the ssh-to-age README.)

To display the age recipient (public key) of this age identity (private key), I used:

midna % nix shell nixpkgs#age
midna 2 % age-keygen -y $HOME/.config/sops/age/keys.txt
age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf

Step 3. Obtain an age recipient for the remote machine

Similarly, I will derive an age recipient from the SSH host key of the remote system:

batchn % cat /etc/ssh/ssh_host_ed25519_key.pub | nix run nixpkgs#ssh-to-age
age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k

Step 4. Configure sops for your git repository

In my git repository (nix-configs), I have one subdirectory per NixOS system, i.e. tree(1) shows:

├── batchn
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
│   ├── secrets
│   │   └── example.yaml
├── wiki
│   ├── configuration.nix
│   ├── disk-config.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── hardware-configuration.nix
│   ├── Makefile
…

In the root of the git repository (next to the batchn directory), I create .sops.yaml like so:

keys:
  - &admin_michael age10e9tt2qwq90y5hvl35dau0sm5cm4qvegtw2a70v7sz5fy99de42s9d5nkf
  - &server_batchn age1wnwfnrqhewjh39pmtyc8zhqw606znskt4h5p9s3pve4apd67gapqj6tr0k
# …more server keys go here…
creation_rules:
  - path_regex: batchn/secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
    - age:
      - *admin_michael
      - *server_batchn

The more systems I manage, the more keys and creation_rules I will need to configure.

The creation rules tell sops which keys to use when encrypting a file. In my setups, I typically use only a single file per system, but I could imagine splitting out some secrets into a separate file if I wanted to collaborate with someone on just one aspect of the system.

Step 5. Manage some secrets with sops

Now that we told sops which recipients to encrypt for, we can decrypt and edit secrets/example.yaml in our configured editor by running:

midna ~/nix-configs/batchn % nix run nixpkgs#sops secrets/example.yaml

The simplest key file contains just one key, for example:

api-key: hello world :)

After saving and exiting your editor, sops will update the encrypted secrets/example.yaml.

Step 6. Configure sops in NixOS

Now, we need to reference the encrypted file in NixOS and enable sops-nix integration to make the decrypted secrets available on the system.

In flake.nix, I added sops-nix to the inputs section and added the NixOS module. I show the entire diff because the places where the lines go are just as important as what the lines say:

--- c/batchn/flake.nix
+++ i/batchn/flake.nix
@@ -1,85 +1,93 @@
 {
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

     disko.url = "github:nix-community/disko";
     # Use the same version as nixpkgs
     disko.inputs.nixpkgs.follows = "nixpkgs";

     stapelbergnix.url = "github:stapelberg/nix";

     zkjnastools.url = "github:stapelberg/zkj-nas-tools";

+    sops-nix = {
+      url = "github:Mic92/sops-nix";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
   };

   outputs =
     {
       nixpkgs,
       disko,
       stapelbergnix,
       zkjnastools,
+      sops-nix,
       ...
     }:
     let
       system = "x86_64-linux";
       pkgs = import nixpkgs {
         inherit system;
         config.allowUnfree = false;
       };
     in
     {
       nixosConfigurations.batchn = nixpkgs.lib.nixosSystem {
         inherit system;
         inherit pkgs;
         modules = [
           disko.nixosModules.disko
           ./configuration.nix
           stapelbergnix.lib.userSettings
           # Use systemd for network configuration
           stapelbergnix.lib.systemdNetwork
           # Use systemd-boot as bootloader
           stapelbergnix.lib.systemdBoot
           # Run prometheus node exporter in tailnet
           stapelbergnix.lib.prometheusNode
           zkjnastools.nixosModules.zkjbackup
+          sops-nix.nixosModules.sops
         ];
       };
       formatter.${system} = pkgs.nixfmt-tree;
     };
 }

Then, in configuration.nix, we tell sops-nix to use the SSH host key as identity, where sops will find our secrets and which secrets sops-nix should realize on the remote system:

  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  sops.defaultSopsFile = ./secrets/example.yaml;
  sops.secrets."api-key" = { };

After deploying, we can access the secret on the running system:

batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Of course, even after rebooting the machine, the secrets remain available without a re-deploy:

batchn ~ % uptime
 22:09:23  up   0:00,  1 user,  load average: 0,32, 0,08, 0,03
batchn ~ % sudo cat /run/secrets/api-key
hello world :)%
batchn ~ %

Usage Examples

Now that we have secrets stored in files under /run/secrets, how can we use these secrets?

The following sections show a few common ways.

Usage Example: command-line flags (ExecStart wrapper)

Let’s assume you have deployed a custom Go server as a systemd service on NixOS as follows, and you want to start managing the cleartext secret passed via the -securecookie_hash_key and -securecookie_block_key command-line flags:

{
  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="some-secret-key" \
          -securecookie_block_key="a-different-secret-key"
      '';
    };
  };
}

With the following sops secrets:

fortuneserver:
    securecookie_hash_key: some-secret-key
    securecookie_block_key: a-different-secret-key

…we need to adjust our NixOS config to read these secret files at runtime. Because the ExecStart directive is interpreted by systemd and not passed through a shell, we use the writeShellScript helper and then just cat the files:

{
  sops.secrets."fortuneserver/securecookie_hash_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };
  sops.secrets."fortuneserver/securecookie_block_key" = {
    owner = "fortuneserver";
    restartUnits = [ "fortuneserver.service" ];
  };

  users.groups.fortuneserver = { };
  users.users.fortuneserver = {
    isSystemUser = true;
    group = "fortuneserver";
  };

  systemd.services.fortuneserver = {
    description = "fortuneserver";
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "fortuneserver";
      Group = "fortuneserver";

      ExecStart = pkgs.writeShellScript "fortuneserver-execstart" ''
        "${pkgs.fortuneserver}/bin/fortuneserver" \
          -securecookie_hash_key="$(cat /run/secrets/fortuneserver/securecookie_hash_key)" \
          -securecookie_block_key="$(cat /run/secrets/fortuneserver/securecookie_block_key)"
      '';
    };
  };
}

Usage Example: environment variable files

What if the service in question does not use command-line flags, but environment variables for configuring secrets? We can put an environment variable file into a sops-managed secret:

translate-fe:
    env: |
        DEEPL_AUTH_KEY=my-deepl-key

…and then we make systemd apply these environment variables from the secrets file:

{
  sops.secrets."translate-fe/env" = {
    owner = "translatefe";
    restartUnits = [ "translate-fe.service" ];
  };

  systemd.services.translate-fe = {
    documentation = [ "https://michael.stapelberg.ch" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "translatefe";

      EnvironmentFile = [ config.sops.secrets."translate-fe/env".path ];

      ExecStart = "${translatefeExecstart}/bin/translate-fe";
    };
  };
}

If you are configuring a NixOS module (instead of declaring a custom service), the option might not always be called EnvironmentFile. For example, for the oauth2-proxy service, you would need to configure the services.oauth2-proxy.keyFile option:

  services.oauth2-proxy = {
    keyFile = config.sops.secrets."oauth2-proxy/env".path;
    enable = true;
    # …
  };

Usage Example: systemd credentials

In the previous examples, we configured the owner of each secret to the user account under which the service is running. But what if there is no such user account, because the service use systemd’s DynamicUser feature?

We can use systemd’s LoadCredential feature! For example, I supply the SMTP password to my Prometheus Alertmanager as follows:

{
  sops.secrets."alertmanager/smtp_pw" = {
    restartUnits = [ "alertmanager.service" ];
  };

  systemd.services.alertmanager.serviceConfig.LoadCredential = [
    "smtp_pw:${config.sops.secrets."alertmanager/smtp_pw".path}"
  ];

  services.prometheus.alertmanager = {
    enable = true;

    configuration = {
      global = {
        smtp_smarthost = "smtp.gmail.com:587";
        smtp_from = "alerts@example.net";
        smtp_auth_username = "alerts@example.net";
        smtp_auth_password_file = "/run/credentials/alertmanager.service/smtp_pw";
      };

      # …remaining config goes here…
    };
  };
}

Usage Example: samba users/passwords

In my blog post “Migrating my NAS from CoreOS/Flatcar Linux to NixOS”, I describe how to configure samba users and passwords (from sops-managed secrets) with an ExecStartPre shell script (which is very similar to the techniques already explained).

Conclusion

Managing secrets as separately-encrypted files in your config repository makes sense to me!

age’s ability to work with SSH keys makes for a really convenient setup, in my opinion. Encrypting secrets for the destination system’s SSH host key feels very elegant.

I hope the examples above are sufficient for you to efficiently configure secrets in NixOS!

Development shells with Nix: four quick examples

2025-07-27T08:50:00+02:00

I wanted to use GoCV for one of my projects (to find and extract paper documents from within a larger scan), without permanently having OpenCV on my system.

This seemed like a good example use-case to demonstrate a couple of Nix commands I like to use, covering quick interactive one-off dev shells to fully declarative, hermetic, reproducible, shareable dev shells.

Notably, you don’t need to use NixOS to run these commands! You can install and use Nix on any Linux system like Debian, Arch, etc., as long as you set a Nix path or use Flakes (see setup).

For comparison: The Debian Way

Before we start looking at Nix, I will show how to get GoCV running on Debian.

Let’s create a minimal Go program which uses a GoCV function like gocv.NewMat(), just to verify that we can compile this program:

package main

import "gocv.io/x/gocv"

func main() {
  gocv.NewMat()
}

If we try to build this on a Debian system, we get:

debian % mkdir -p /tmp/minimal
debian % cd /tmp/minimal

debian % cat > minimal.go <<'EOT'
package main
import "gocv.io/x/gocv"
func main() { gocv.NewMat(); }
EOT

debian % go mod init minimal
go: creating new go.mod: module minimal
go: to add module requirements and sums:
	go mod tidy

debian % go mod tidy
go: finding module for package gocv.io/x/gocv
go: downloading gocv.io/x/gocv v0.41.0
go: found gocv.io/x/gocv in gocv.io/x/gocv v0.41.0

debian % go build
# gocv.io/x/gocv
# [pkg-config --cflags  -- opencv4]
Package opencv4 was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv4.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv4', required by 'virtual:world', not found

On Debian, we can install OpenCV as follows:

debian % sudo apt install libopencv-dev

[…]

Summary:
  Upgrading: 7, Installing: 512, Removing: 0, Not Upgrading: 27
  Download size: 367 MB
  Space needed: 1590 MB / 281 GB available

Continue? [Y/n]

Saying “yes” to this prompt downloads and installs over 500 packages (takes a few minutes).

Now the build works:

debian % go build
debian % file minimal
minimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), […]

…but we have over 500 extra packages on our system that will now need to be updated in all eternity, therefore I would like to separate this one-off experiment from my usual system.

We could use Docker to start a Debian container and work inside that container, but, depending on the task, this can be cumbersome precisely because it’s a separate environment. For this example, I would need to specify a volume mount to make my input files available to the Docker container, and I would need to set up environment variables before programs inside the Docker container can open graphical windows on the host…

Let’s look at how we can use Nix to help us with that!

Setup: Nix-on-Debian (or Nix-on-Arch, or…)

Users of NixOS can skip this section, as NixOS systems include a ready-to-use Nix.

Before you can try the examples on your own computer, you need to complete these three steps:

Install Nix
Enable Flakes
Set a Nix path

Step 1: Install Nix

Users of Debian, Arch, Fedora, or other Linux systems first need to install Nix. Luckily, Nix is available for many popular Linux distributions:

Debian ships nix-setup-systemd
Arch Linux packages nix and provides documentation on the Nix Arch Wiki page. In practice, I installed the package and configured a couple of nixbld users.
More generally, there are Nix builds (rpm, deb, pacman) available for a number of distributions: https://github.com/nix-community/nix-installers

Step 2: Enable Flakes

Nix flakes are “a generic way to package Nix artifacts”.

Examples 3 and 4 use Nix flakes to pin dependencies, so we need to enable Nix flakes.

Step 3: Set a Nix path

For example 1 and 2, we want to use the Nix expression import .

On NixOS, this expression will follow the system version, meaning if you use import on a NixOS 25.05 installation, that will reference nixpkgs in version nixos-25.05.

On other Linux systems, you’ll see an error message like this:

debian-server % nix-shell -p pkg-config opencv
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

       at «string»:1:25:

            1| {...}@args: with import  args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (pkg-config) (opencv) ]; } ""
             |                         ^
(use '--show-trace' to show detailed location information)

We need to tell Nix which version of nixpkgs to use by setting the Nix search path:

debian-server % export NIX_PATH=nixpkgs=channel:nixos-25.05
debian-server % nix-shell -p pkg-config opencv
[nix-shell:/tmp/opencv]#

Alright! Now we are set up. Let’s jump into the first example!

Example 1: Interactive one-offs: nix-shell

Nix provides a middle-ground between installing OpenCV on your system (apt install like in the example above) and installing OpenCV in a separate Docker container: Nix can make available OpenCV without permanently installing it.

We can run nix-shell(1) to start a bash shell in which the specified packages are available. To successfully build Go code that uses GoCV, we need to have OpenCV available:

% nix-shell -p pkg-config opencv
these 194 paths will be fetched (175.80 MiB download, 764.10 MiB unpacked):
  /nix/store/ig2nk0hsha9xaailhaj69yv677nv95q4-abseil-cpp-20210324.2
  /nix/store/yw5xqn8lqinrifm9ij80nrmf0i6fdcbx-alsa-lib-1.2.13
[…]

[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

In case you were wondering: Yes, we do need to specify pkg-config in this nix-shell command explicitly, otherwise running pkg-config will run the host version (outside the dev shell), which cannot find opencv4.pc.

Example 2: nix-shell config file: shell.nix

Once we have a combination of packages that work for our project (in our example, just pkg-config and opencv), we can create a shell.nix (in any directory, but usually in the root of a project) which nix-shell (without the -p flag) will read:

{
  pkgs ? import  { },
}:
pkgs.mkShell {
  packages = with pkgs; [
    # Explicitly list pkg-config so that mkShell will arrange
    # for the PKG_CONFIG_PATH to find the .pc files.
    pkg-config
    opencv
  ];
}

…and then, we just run nix-shell:

% nix-shell
[nix-shell:/tmp/opencv]$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

If you’re curious, here are a couple of documentation pointers regarding the boilerplate around the list of packages:

Line 1 to 3 declare a function with an argument set — this is the required structure for nix-shell to be able to call your shell.nix file.
pkgs.mkShell is a convenience helper for use with nix-shell.
The with pkgs; part allows us to write opencv instead of pkgs.opencv.

By the way: With the nixd language server, editors with LSP support can show the versions that packages resolve to, point out your spelling mistakes, or provide features like “jump to definition”.

For example, in this screenshot, I was editing shell.nix in Emacs and was curious how the Nix source of the opencv package looked like. By pressing M-. (xref-find-definitions) with “point” over opencv, I got to opencv/4.x.nix in my local Nix store:

Example 3: Hermetic, pinned devShells: Nix Flakes

The previous examples used nixpkgs from your system (or Nix path), which means you don’t need to change the .nix file when you upgrade your system — depending on the use-case, I see this behavior as either convenient or terrifying.

For use-cases where it is important that the .nix file is built exactly the same way, no matter what version the surrounding OS uses, we can use Nix Flakes to build in a hermetic way, with dependency versions pinned in the flake.lock file.

A flake.nix contains the same mkShell expression as above, but declares structure around it: The mkShell expression goes into the outputs.devShells.x86_64-linux.default attribute and the inputs attribute contains Flake references that are available to this build:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";

  outputs =
    { self, nixpkgs }:
    {
      devShells.x86_64-linux.default =
        let
          pkgs = nixpkgs.legacyPackages.x86_64-linux;
        in
        pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
    };
}

By the way: Despite the name, it is a best practice to use nixpkgs.legacyPackages, which conceptually provides a single import nixpkgs result (for efficiency).

Now, I can use nix develop to get a shell with OpenCV:

% nix develop
michael@midna$ pkg-config --cflags opencv4
-I/nix/store/mh5b1dx2ifv4jkp9a8lgssxwhzxssb96-opencv-4.11.0/include/opencv4

The first nix develop run creates a flake.lock file, so running nix develop later will get us exactly the same environment. To update to newer versions, use nix flake update.

Tip: Instead of a shell, nix develop --command=emacs is also a useful variant.

Example 4: Making the Flake system-independent

Unfortunately, the above flake.nix hard-codes x86_64-linux, so it will not be usable on, say, an aarch64-linux (ARM) computer, or on a x86_64-darwin (Mac).

Having to explicitly specify the system by default is a long-standing criticism of Nix Flakes.

There are a number of workarounds. For example, we can use numtide/flake-utils and refactor our flake.nix to use its eachDefaultSystem convenience function:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        formatter = pkgs.nixfmt-tree;
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            # Explicitly list pkg-config so that mkShell will arrange
            # for the PKG_CONFIG_PATH to find the .pc files.
            pkg-config
            opencv
          ];
        };
      }
    );
}

Or we could use numtide/blueprint, its spiritual successor.

LucPerkins’s dev-templates have effectively inlined a version of this technique.

For a solution that isn’t part of Nix, but Nix-adjacent: devenv is a separate tool that is built on Nix (no longer using the CppNix implementation, but tvix actually), but with its own .nix files.

Tip: Keeping packages around

If you notice that nix develop or similar commands fetch packages despite the flake.lock not having changed, you can install the Flake into your profile to declare it as a gcroot to Nix:

% nix profile install .#devShells.x86_64-linux.default

But wait, isn’t that getting us into the same state as with The Debian Way? No! While OpenCV will remain available indefinitely if you install the flake into your profile, there still is a layer of separation: Within your system, OpenCV isn’t available, only when you start a development shell with nix-shell or nix develop.

Conclusion

How do the four examples above compare? Here’s an overview:

Example	Boilerplate	Pinned?	System-dependent?
Ex 1: `nix-shell -p …`	😊	no	no
Ex 2: `shell.nix`	🙂	no	no
Ex 3: `flake.nix`	😲	yes	yes
Ex 4: system-independent `flake.nix`	🤨	yes	no

For personal one-off experiments, I use nix-shell.

Once the experiment works, I typically want to pin the dependencies, so I use a flake.nix.

If this is software that isn’t just versioned, but also published (or worked on with multiple people/systems), I go through the effort of making it a system-independent flake.nix.

I hope in the future, it will become easier to write a system-independent flake.

Despite the rough edges, I appreciate the reproducibility and control that Nix gives me!

Migrating my NAS from CoreOS/Flatcar Linux to NixOS

2025-07-13T08:17:00+02:00

In this article, I want to show how to migrate an existing Linux server to NixOS — in my case the CoreOS/Flatcar Linux installation on my Network Attached Storage (NAS) PC.

I will show in detail how the previous CoreOS setup looked like (lots of systemd units starting Docker containers), how I migrated it into an intermediate state (using Docker on NixOS) just to get things going, and finally how I migrated all units from Docker to native NixOS modules step-by-step.

If you haven’t heard of NixOS, I recommend you read the first page of the NixOS website to understand what NixOS is and what sort of things it makes possible.

The target audience of this blog post is people interested in trying out NixOS for the use-case of a NAS, who like seeing examples to understand how to configure a system.

You can apply these examples by first following my blog post “How I like to install NixOS (declaratively)”, then making your way through the sections that interest you. If you prefer seeing the full configuration, skip to the conclusion.

Context/History

Over the last decade, I used a number of different operating systems for my NAS needs. Here’s an overview of the 2 NAS systems storage2 and storage3:

Year	storage2	storage3	Details (blog post)
2013	Debian on qnap	Debian on qnap	Wake-On-LAN with Debian on a qnap TS-119P2+
2016	CoreOS on PC	CoreOS on PC	Gigabit NAS (running CoreOS)
2023	CoreOS on PC	Ubuntu+ZFS on PC	My all-flash ZFS NAS build
2025	NixOS on PC	Ubuntu+ZFS on PC	→ you are here ←
?	NixOS on PC	NixOS+ZFS on PC	Converting more PCs to NixOS seems inevitable ;)

My NAS Software Requirements

(This post is only about software! For my usage patterns and requirements regarding hardware selection, see “Design Goals” in my My all-flash ZFS NAS build post (2023).)
Remote management: I really like the model of having the configuration of my network storage builds version-controlled and managed on my main PC. It’s a nice property that I can regain access to my backup setup by re-installing my NAS from my PC within minutes.
Automated updates, with easy rollback: Updating all my installations manually is not my idea of a good time. Hence, automated updates are a must — but when the update breaks, a quick and easy path to recovery is also a must.
- CoreOS/Flatcar achieved that with an A/B updating scheme (update failed? boot the old partition), whereas NixOS achieves that with its concept of a “generation” (update failed? select the old generation), which is finer-grained.

Why migrate from CoreOS/Flatcar to NixOS?

When I started using CoreOS, Docker was pretty new technology. I liked that using Docker containers allowed you to treat services uniformly — ultimately, they all expose a port of some sort (speaking HTTP, or Postgres, or…), so you got the flexibility to run much more recent versions of software on a stable OS, or older versions in case an update broke something.

Over a decade later, Docker is established tech. People nowadays take for granted the various benefits of the container approach.

So, here’s my list of reasons why I wasn’t satisfied with Flatcar Linux anymore.

R1. cloud-init is deprecated

The CoreOS cloud-init project was deprecated at some point in favor of Ignition, which is clearly more powerful, but also more cumbersome to get started with as a hobbyist. As far as I can tell, I must host my config at some URL that I then provide via a kernel parameter. The old way of just copying a file seems to no longer be supported.

Ignition also seems less convenient in other ways: YAML is no longer supported, only JSON, which I don’t enjoy writing by hand. Also, the format seems to change quite a bit.

As a result, I never made the jump from cloud-init to Ignition, and it’s not good to be reliant on a long-deprecated way to use your OS of choice.

R2. Container Bitrot

At some point, I did an audit of all my containers on the Docker Hub and noticed that most of them were quite outdated. For a while, Docker Hub offered automated builds based on a Dockerfile obtained from GitHub. However, automated builds now require a subscription, and I will not accept a subscription just to use my own computers.

R3. Dependency on a central service

If Docker at some point ceases operation of the Docker Hub, I am unable to deploy software to my NAS. This isn’t a very hypothetical concern: In 2023, Docker Hub announced the end of organizations on the Free tier and then backpedaled after community backlash.

Who knows how long they can still provide free services to hobbyists like myself.

R4. Could not try Immich on Flatcar

The final nail in the coffin was when I noticed that I could not try Immich on my NAS system! Modern web applications like Immich need multiple Docker containers (for Postgres, Redis, etc.) and hence only offer Docker Compose as a supported way of installation.

Unfortunately, Flatcar does not include Docker Compose.

I was not in the mood to re-package Immich for non-Docker-Compose systems on an ongoing basis, so I decided that a system on which I can neither run software like Immich directly, nor even run Docker Compose, is not sufficient for my needs anymore.

Reason Summary

With all of the above reasons, I would have had to set up automated container builds, run my own central registry and would still be unable to run well-known Open Source software like Immich.

Instead, I decided to try NixOS again (after a 10 year break) because it seems like the most popular declarative solution nowadays, with a large community and large selection of packages.

How does NixOS compare for my situation?

Same: I also need to set up an automated job to update my NixOS systems.
- I already have such a job for updating my gokrazy devices.
- Docker push is asynchronous: After a successful push, I still need extra automation for pulling the updated containers on the target host and restarting the affected services, whereas NixOS includes all of that.
Better: There is no central registry. With NixOS, I can push the build result directly to the target host via SSH.
Better: The corpus of available software in NixOS is much larger (including Immich, for example) and the NixOS modules generally seem to be expressed at a higher level of abstraction than individual Docker containers, meaning you can configure more features with fewer lines of config.

Prototyping in a VM

My NAS setup needs to work every day, so I wanted to prototype my desired configuration in a VM before making changes to my system. This is not only safer, it also allows me to discover any roadblocks, and what working with NixOS feels like without making any commitments.

I copied my NixOS configuration from a previous test installation (see “How I like to install NixOS (declaratively)”) and used the following command to build a VM image and start it in QEMU:

nix build .#nixosConfigurations.storage2.config.system.build.vm

export QEMU_NET_OPTS=hostfwd=tcp::2222-:22
export QEMU_KERNEL_PARAMS=console=ttyS0
./result/bin/run-nixplay-vm

The configuration instructions below can be tried out in this VM, and once you’re happy enough with what you have, you can repeat the steps on the actual machine to migrate.

Migrating

For the migration of my actual system, I defined the following milestones that should be achievable within a typical session of about an hour (after prototyping them in a VM):

M1. Install NixOS
M2. Set up remote disk unlock
M3. Set up Samba for access
M4. Set up SSH/rsync for backups
Everything extra is nice-to-have and could be deferred to a future session on another day.

In practice, this worked out exactly as planned: the actual installation of NixOS and setting up my config to milestone M4 took a little over one hour. All the other nice-to-haves were done over the following days and weeks as time permitted.

Tip: After losing data due to an installer bug in the 2000s, I have adopted the habit of physically disconnecting all data disks (= pulling out the SATA cable) when re-installing the system disk.

M1. Install NixOS

After following “How I like to install NixOS (declaratively)”, this is my initial configuration.nix:

{ modulesPath, lib, pkgs, ... }:

{
  imports =
    [
      (modulesPath + "/installer/scan/not-detected.nix")
      ./hardware-configuration.nix
      ./disk-config.nix
    ];

  # Adding michael as trusted user means
  # we can upgrade the system via SSH (see Makefile).
  nix.settings.trusted-users = [ "michael" "root" ];
  # Clean the Nix store every week.
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 7d";
  };

  boot.loader.systemd-boot = {
    enable = true;
    configurationLimit = 10;
  };
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "storage2";
  time.timeZone = "Europe/Zurich";

  # Use systemd for networking
  services.resolved.enable = true;
  networking.useDHCP = false;
  systemd.network.enable = true;

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
  };

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  users.mutableUsers = false;
  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "networkmanager" "wheel" ];
    initialPassword = "secret";  # XXX: change!
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
  ];

  programs.zsh.enable = true;

  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

All following sections describe changes within this configuration.nix.

All devices in my home network obtain their IP address via DHCP. If I want to make an IP address static, I configure it accordingly on my router.

My NAS PCs have one specialty with regards to IP addressing: They are reachable via IPv4 and IPv6, and the IPv6 address can be derived from the IPv4 address.

Hence, I changed the systemd-networkd configuration from above such that it configures a static IPv6 address in a dynamically configured IPv6 network:

  systemd.network.networks."10-e" = {
    matchConfig.Name = "e*";  # enp9s0 (10G) or enp8s0 (1G)
    networkConfig = {
      IPv6AcceptRA = true;
      DHCP = "yes";
    };
    ipv6AcceptRAConfig = {
      Token = "::10:0:0:252";
    };
  };

✅ This fulfills milestone M1.

M2. Set up remote disk unlock

To unlock my encrypted disks on boot, I have a custom systemd service unit that uses wget(1) and cryptsetup(8) to split the key file between the NAS and a remote server (= an attacker needs both pieces to unlock).

With CoreOS/Flatcar, my cloud-init configuration looked as follows:

coreos:
  units:
    - name: unlock.service
      command: start
      content: |
        [Unit]
        Description=unlock hard drive
        Wants=network.target
        After=systemd-networkd-wait-online.service
        Before=samba.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # Wait until the host is actually reachable.
        ExecStart=/bin/sh -c "c=0; while [ $c -lt 5 ]; do /bin/ping6 -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"
        ExecStart=/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/nascrypto) | /sbin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"
        ExecStart=/bin/sh -c "vgchange -ay"
        ExecStart=/bin/mount /dev/mapper/data-data /srv

write_files:
  - path: /etc/ssl/certs/r.zekjur.net.crt
    content: |
      -----BEGIN CERTIFICATE-----
      MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
      […]
      -----END CERTIFICATE-----

I converted it into the following NixOS configuration:

  systemd.services.unlock = {
    wantedBy = [ "multi-user.target" ];
    description = "unlock hard drive";
    wants = [ "network.target" ];
    after = [ "systemd-networkd-wait-online.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = "yes";
      ExecStart = [
        # Wait until the host is actually reachable.
        ''/bin/sh -c "c=0; while [ $c -lt 5 ]; do ${pkgs.iputils}/bin/ping -n -c 1 r.zekjur.net && break; c=$((c+1)); sleep 1; done"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNF0T205183F_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdb2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0T205183F S5SSNF0T205183F_crypt"''

        ''/bin/sh -c "[ -e \"/dev/mapper/S5SSNJ0T205991B_crypt\" ] || (echo -n my_local_secret && ${pkgs.wget}/bin/wget --retry-connrefused --ca-directory=/dev/null --ca-certificate=/etc/ssl/certs/r.zekjur.net.crt -qO - https://r.zekjur.net:8443/sdc2_crypt) | ${pkgs.cryptsetup}/bin/cryptsetup --key-file=- luksOpen /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNJ0T205991B S5SSNJ0T205991B_crypt"''

        ''/bin/sh -c "${pkgs.lvm2}/bin/vgchange -ay"''
        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
      ];
    };

  };

We’ll also need to store the custom TLS certificate file on disk. For that, we can use the environment. configuration:

  environment.etc."ssl/certs/r.zekjur.net.crt".text = ''
-----BEGIN CERTIFICATE-----
MIID8TCCAlmgAwIBAgIRAPWwvYWpoH+lGKv6rxZvC4MwDQYJKoZIhvcNAQELBQAw
[…]
-----END CERTIFICATE-----
'';

The references like ${pkgs.wget} will be replaced with a path to the Nix store (→ nix.dev documentation). On CoreOS/Flatcar, I was limited to using just the (minimal set of) software included in the base image, or I had to reach for Docker. On NixOS, we can use all packages available in nixpkgs.

After deploying and rebooting, I can access my unlocked disk under /srv! 🎉

% df -h /srv
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/data-data   15T   14T  342G  98% /srv

When listing my files, I noticed that the group id was different between my old system and the new system. This can be fixed by explicitly specifying the desired group id:

  users.groups.michael = {
    gid = 1000;  # for consistency with storage3
  };

✅ M2 is complete.

M3. Set up Samba for access

Whereas I want to configure remote disk unlock at the systemd service level, for Samba I want to use Docker: I wanted to first transfer my old (working) Docker-based setups as they are, and only later convert them to Nix.

We enable the Docker NixOS module which sets up the daemons that Docker needs and whatever else is needed to make it work:

  virtualisation.docker.enable = true;

This is already sufficient for other services to use Docker, but I also want to be able to run the docker command interactively for debugging. Therefore, I added docker to systemPackages:

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    docker
  ];

After deploying this configuration, I can run docker run -ti debian to verify things work.

The cloud-init version of samba looked like this:

coreos:
  units:
    - name: samba.service
      command: start
      content: |
        [Unit]
        Description=samba server
        After=docker.service unlock.mount
        Requires=docker.service unlock.mount

        [Service]
        Restart=always
        StartLimitInterval=0

        # Always pull the latest version (bleeding edge).
        ExecStartPre=-/usr/bin/docker pull stapelberg/docker-samba:latest

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ExecStartPre=-/usr/bin/docker kill smb
        ExecStartPre=-/usr/bin/docker rm smb
        ExecStartPre=-/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf'
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared
        ExecStartPre=/usr/bin/docker rm smb-prep
        ExecStartPre=/usr/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"
        ExecStartPre=/usr/bin/docker commit smb-prep smb-prepared

        ExecStart=/usr/bin/docker run \
          -p 137:137 \
          -p 138:138 \
          -p 139:139 \
          -p 445:445 \
          --tmpfs=/run \
          -v /srv/data:/srv/data \
          --name smb \
          -t \
          smb-prepared \
            /usr/sbin/smbd --foreground --debug-stdout --no-process-group

We can translate this 1:1 to NixOS:

  systemd.services.samba = {
    wantedBy = [ "multi-user.target" ];
    description = "samba server";
    after = [ "unlock.service" ];
    requires = [ "unlock.service" ];
    serviceConfig = {
      Restart = "always";
      StartLimitInterval = 0;
      ExecStartPre = [
        # Always pull the latest version.
        ''-${pkgs.docker}/bin/docker pull stapelberg/docker-samba:latest''

        # Set up samba users (cannot be done in the (public) Dockerfile because
        # users/passwords are sensitive information).
        ''-${pkgs.docker}/bin/docker kill smb''
        ''-${pkgs.docker}/bin/docker rm smb''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep stapelberg/docker-samba sh -c 'adduser --quiet --disabled-password --gecos "" --uid 29901 michael && sed -i "s,\\[global\\],[global]\\nserver multi channel support = yes\\naio read size = 1\\naio write size = 1,g" /etc/samba/smb.conf' ''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
        ''-${pkgs.docker}/bin/docker rm smb-prep''
        ''-${pkgs.docker}/bin/docker run --name smb-prep smb-prepared /bin/sh -c "echo \"secret\nsecret\n" | tee - | smbpasswd -a -s michael"''
        ''-${pkgs.docker}/bin/docker commit smb-prep smb-prepared''
      ];

      ExecStart = ''-${pkgs.docker}/bin/docker run \
           -p 137:137 \
           -p 138:138 \
           -p 139:139 \
           -p 445:445 \
           --tmpfs=/run \
           -v /srv/data:/srv/data \
           --name smb \
           -t \
           smb-prepared \
             /usr/sbin/smbd --foreground --debug-stdout --no-process-group
             '';
    };
  };
}

✅ Now I can manage my files over the network, which completes M3!

M4. Set up SSH/rsync for backups

For backing up data, I use rsync over SSH. I restrict this SSH access to run only rsync commands by using rrsync (in a Docker container). To configure the SSH authorized_keys(5) , we set:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  };

✅ A successful test backup run completes milestone M4!

Nice-to-haves

N1. Prometheus Node Exporter

I like to monitor all my machines with Prometheus (and Grafana). For network connectivity and authentication, I use the Tailscale mesh VPN.

To install Tailscale, I enable its NixOS module and make the tailscale command available:

  services.tailscale.enable = true;
  environment.systemPackages = with pkgs; [ tailscale ];

After deploying, I run sudo tailscale up and open the login link in my browser.

The Prometheus Node Exporter can also easily be enabled through its NixOS module:

  services.prometheus.exporters.node = {
    enable = true;
    listenAddress = "storage2.example.ts.net";
  };

However, this isn’t reliable yet: When Tailscale’s startup takes a while during system boot, the Node Exporter might burn through its entire restart budget when it cannot listen on the Tailscale IP address yet. We can enable indefinite restarts for the service to eventually come up:

  systemd.services."prometheus-node-exporter" = {
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      RestartSec = 1;
    };
  };

N2. Reliable mounting

While migrating my setup, I noticed that calling mount(8) from unlock.service directly is not reliable, and it’s better to let systemd manage the mounting:

  fileSystems."/srv" = {
    device = "/dev/mapper/data-data";
    fsType = "ext4";
    options = [
      "nofail"
      "x-systemd.requires=unlock.service"
    ];
  };

Afterwards, I could just remove the mount(8) call from unlock.service:

@@ -247,7 +247,10 @@ fry/U6A=
         ''/bin/sh -c "${pkgs.lvm2.bin}/bin/vgchange -ay"''
-        ''/run/wrappers/bin/mount /dev/mapper/data-data /srv''
+        # Let systemd mount /srv based on the fileSystems./srv
+        # declaration to prevent race conditions: mount
+        # might not succeed while the fsck is still in progress,
+        # for example, which otherwise makes unlock.service fail.
       ];
     };

In systemd services, I can now depend on the /srv mount unit:

  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N3. nginx-healthz

To save power, I turn off my NAS when they are not in use.

My backup orchestration uses Wake-on-LAN to start a wakeup and needs to wait until the NAS is fully booted up and has mounted its /srv mount before it can start backup jobs.

For this purpose, I have configured a web server (without any files) that depends on the /srv mount. So, once the web server responds to HTTP requests, we know /srv is mounted.

The cloud-init config looked as follows:

coreos:
  units:
    - name: healthz.service
      command: start
      content: |
        [Unit]
        Description=nginx for /srv health check
        Wants=network.target
        After=srv.mount
        Requires=srv.mount
        StartLimitInterval=0

        [Service]
        Restart=always
        ExecStartPre=/bin/sh -c 'systemctl is-active docker.service'
        ExecStartPre=/usr/bin/docker pull nginx:1
        ExecStartPre=-/usr/bin/docker kill nginx-healthz
        ExecStartPre=-/usr/bin/docker rm -f nginx-healthz
        ExecStart=/usr/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1

The Docker version (ported from Flatcar Linux) looks like this:

  systemd.services.healthz = {
    description = "nginx for /srv health check";
    wants = [ "network.target" ];
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''/bin/sh -c 'systemctl is-active docker.service' ''
        ''-${pkgs.docker}/bin/docker pull nginx:1''
        ''-${pkgs.docker}/bin/docker kill nginx-healthz''
        ''-${pkgs.docker}/bin/docker rm -f nginx-healthz''
      ];

      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
            --name nginx-healthz \
            --publish 10.0.0.252:8200:80 \
            --log-driver=journald \
            nginx:1
        ''
      ];
    };
  };

This configuration gets a lot simpler when migrating it from Docker to NixOS:

  # Signal readiness on HTTP port 8200 once /srv is mounted:
  networking.firewall.allowedTCPPorts = [ 8200 ];
  services.caddy = {
    enable = true;
    virtualHosts."http://10.0.0.252:8200".extraConfig = ''
      respond "ok"
    '';
  };
  systemd.services.caddy = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

N4. NixOS Jellyfin

The Docker version (ported from Flatcar Linux) looks like this:

  networking.firewall.allowedTCPPorts = [ 4414 8096 ];
  systemd.services.jellyfin = {
    wantedBy = [ "multi-user.target" ];
    description = "jellyfin";
    after = [ "docker.service" "srv.mount" ];
    requires = [ "docker.service" "srv.mount" ];
    startLimitIntervalSec = 0;
    serviceConfig = {
      Restart = "always";
      ExecStartPre = [
        ''-${pkgs.docker}/bin/docker pull lscr.io/linuxserver/jellyfin:latest''
        ''-${pkgs.docker}/bin/docker rm jellyfin''
      ];
      ExecStart = [
        ''-${pkgs.docker}/bin/docker run \
          --rm \
          --net=host \
          --name=jellyfin \
          -e TZ=Europe/Zurich \
          -v /srv/jellyfin/config:/config \
          -v /srv/data/movies:/data/movies:ro \
          -v /srv/data/series:/data/series:ro \
          -v /srv/data/mp3:/data/mp3:ro \
          lscr.io/linuxserver/jellyfin:latest
        ''
      ];
    };
  };

As before, when using jellyfin from NixOS, the configuration gets simpler:

  services.jellyfin = {
    enable = true;
    openFirewall = true;
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

For a while, I had also set up compatibility symlinks that map the old location (/data/movies, inside the Docker container) to the new location (/srv/data/movies), but I encountered strange issues in Jellyfin and ended up just re-initializing my whole Jellyfin state. While the required configuration had more lines, I found it neat to move it into its own file, so here is how to do that:

Remove the lines above from configuration.nix and move them into jellyfin.nix:

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}:

{
  services.jellyfin = {
    enable = true;
    openFirewall = true;
    dataDir = "/srv/jellyfin";
    cacheDir = "/srv/jellyfin/config/cache";
  };
  systemd.services.jellyfin = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };
}

Then, in configuration.nix, add jellyfin.nix to the imports:

   imports = [
     ./hardware-configuration.nix
     ./jellyfin.nix
   ];

N5. NixOS samba

To use Samba from NixOS, I replaced my systemd.services.samba config from M3 with this:

  services.samba = {
    enable = true;
    openFirewall = true;
    settings = {
      "global" = {
        "map to guest" = "bad user";
      };
      "data" = {
        "path" = "/srv/data";
        "comment" = "public data";
        "read only" = "no";
        "create mask" = "0775";
        "directory mask" = "0775";
        "guest ok" = "yes";
      };
    };
  };
  system.activationScripts.samba_user_create = ''
      smb_password="secret"
      echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s michael
    '';

Note: Setting the samba password in the activation script works for small setups, but if you want to keep your samba passwords out of the Nix store, you’ll need to use a different approach. On a different machine, I use sops-nix to manage secrets and found that refactoring the smbpasswd call like so works reliably:

let
  setPasswords = pkgs.writeShellScript "samba-set-passwords" ''
    set -euo pipefail
    for user in michael; do
        smb_password="$(cat /run/secrets/samba_passwords/$user)"
        echo -e "$smb_password\n$smb_password\n" | ${lib.getExe' pkgs.samba "smbpasswd"} -a -s $user
    done
  '';
in
 {
  # …
  services.samba = {
    # …as above…
  }

  systemd.services.samba-smbd.serviceConfig.ExecStartPre = [
    "${setPasswords}"
  ];

  sops.secrets."samba_passwords/michael" = {
    restartUnits = [ "samba-smbd.service" ];
  };
}

I also noticed that NixOS does not create a group for each user by default, but I am used to managing my permissions like that. We can easily declare a group like so:

  users.groups.michael = {
    gid = 1000; # for consistency with storage3
  };
  users.users.michael = {
    extraGroups = [
      "wheel" # Enable ‘sudo’ for the user.
      "docker"
      # By default, NixOS does not add users to their own group:
      # https://github.com/NixOS/nixpkgs/issues/198296
      "michael"
    ];
  };

N6. NixOS rrsync

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/midna:/srv/backup/midna stapelberg/docker-rsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

To use rrsync from NixOS, I changed the configuration like so:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.rrsync}/bin/rrsync /srv/backup/midna" ssh-rsa AAAAB3Npublickey root@midna''
  ];

N7. sync.pl script

The Docker version (ported from Flatcar Linux) looks like this:

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${pkgs.docker}/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/data:/srv/data -v /root/.ssh:/root/.ssh:ro -v /etc/ssh:/etc/ssh:ro -v /etc/static/ssh:/etc/static/ssh:ro -v /nix/store:/nix/store:ro stapelberg/docker-sync",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

I wanted to stop managing the following Dockerfile to ship sync.pl:

FROM debian:stable

# Install full perl for Data::Dumper
RUN apt-get update \
    && apt-get install -y rsync ssh perl

ADD sync.pl /usr/bin/

ENTRYPOINT ["/usr/bin/sync.pl"]

To get rid of the Docker container, I translated the sync.pl file into a Nix expression that writes the sync.pl Perl script to the Nix store:

{ pkgs }:

# For string literal escaping rules (''${), see:
# https://nix.dev/manual/nix/2.26/language/string-literals#string-literals

# For writers.writePerlBin, see https://wiki.nixos.org/wiki/Nix-writers

pkgs.writers.writePerlBin "syncpl" { libraries = []; } ''
# This script is run via ssh from dornröschen.
use strict;
use warnings;
use Data::Dumper;

if (my ($destination) = ($ENV{SSH_ORIGINAL_COMMAND} =~ /^([a-z0-9.]+)$/)) {
    print STDERR "rsync version: " . `${pkgs.rsync}/bin/rsync --version` . "\n\n";
    my @rsync = (
        "${pkgs.rsync}/bin/rsync",
        "-e",
        "ssh",
        "--max-delete=-1",
        "--verbose",
        "--stats",
        # Intentionally not setting -X for my data sync,
        # where there are no full system backups; mostly media files.
        "-ax",
        "--ignore-existing",
        "--omit-dir-times",
        "/srv/data/",
        "''${destination}:/",
    );
    print STDERR "running: " . Dumper(\@rsync) . "\n";
    exec @rsync;
} else {
    print STDERR "Could not parse SSH_ORIGINAL_COMMAND.\n";
}
''

I can then reference this file by importing it in my configuration.nix and pointing it to the pkgs expression of my NixOS configuration:

{ modulesPath, lib, pkgs, ... }:

let
  syncpl = import ./syncpl.nix { pkgs = pkgs; };
in {
  imports = [ ./hardware-configuration.nix ];

  users.users.root.openssh.authorizedKeys.keys = [
    ''command="${syncpl}/bin/syncpl",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3Npublickey sync@dr''
  ];

  # For interactive usage (when debugging):
  environment.systemPackages = [ syncpl ];

  # …
}

This works, but is it the best approach? Here are some thoughts:

By managing this script in a Nix expression, I can no longer use my editor’s Perl support.
- I could probably also keep sync.pl as a separate file and use string interpolation in my Nix expression to inject an absolute path to the rsync binary into the script.
Another alternative would be to add a wrapper script to my Nix expression which ensures that $PATH contains rsync and then the script wouldn’t need an absolute path anymore.
For small glue scripts like this one, I consider it easier to manage the contents “inline” in the Nix expression, because it means one fewer file in my config directory.

N8. Sharing configs

I want to configure all my NixOS systems such that my user settings are identical everywhere.

To achieve that, I can extract parts of my configuration.nix into a user-settings.nix and then declare an accompanying flake.nix that provides this expression as an output.

After publishing these files in a git repository, I can reference said repository in my flake.nix:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";
    stapelbergnix.url = "github:stapelberg/nix";
  };

  outputs =
    {
      self,
      nixpkgs,
	  stapelbergnix,
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.storage2 = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          ./configuration.nix
          stapelbergnix.lib.userSettings
          # Not on this machine; We have our own networking config:
          # stapelbergnix.lib.systemdNetwork
          # Use systemd-boot as bootloader
          stapelbergnix.lib.systemdBoot
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Everything declared in the user-settings.nix can now be removed from configuration.nix!

N9. Trying immich!

One of the motivating reasons for switching away from CoreOS/Flatcar was that I couldn’t try Immich, so let’s give it a shot on NixOS:

  services.immich = {
    enable = true;
    host = "10.0.0.252";
    port = 2283;
    openFirewall = true;
    mediaLocation = "/srv/immich";
  };

  # Because /srv is a separate file system, we need to declare:
  systemd.services."immich-server" = {
    unitConfig.RequiresMountsFor = [ "/srv" ];
    wantedBy = [ "srv.mount" ];
  };

Conclusion

You can find the full configuration directory on GitHub.

I am pretty happy with this NixOS setup! Previously (with CoreOS/Flatcar), I could declaratively manage my base system, but had to manage tons of Docker containers in addition. With NixOS, I can declaratively manage everything (or as much as makes sense).

Custom configuration like my SSH+rsync-based backup infrastructure can be expressed cleanly, in one place, and structured at the desired level of abstraction/reuse.

If you’re considering managing at least one other system with NixOS, I would recommend it! One of my follow-up projects is to convert storage3 (my other NAS build) from Ubuntu Server to NixOS as well to cut down on manual management. Being able to just copy the entire config to set up another system, or try out an idea in a throwaway VM, is just such a nice workflow 🥰

…but if you have just a single system to manage, probably all of this is too complicated.

How I like to install NixOS (declaratively)

2025-06-01T08:20:43+02:00

For one of my network storage PC builds, I was looking for an alternative to Flatcar Container Linux and tried out NixOS again (after an almost 10 year break). There are many ways to install NixOS, and in this article I will outline how I like to install NixOS on physical hardware or virtual machines: over the network and fully declaratively.

Introduction: Declarative?

The term declarative means that you describe what should be accomplished, not how. For NixOS, that means you declare what software you want your system to include (add to config option environment.systemPackages, or enable a module) instead of, say, running apt install.

A nice property of the declarative approach is that your system follows your configuration, so by reverting a configuration change, you can cleanly revert the change to the system as well.

I like to manage declarative configuration files under version control, typically with Git.

When I originally set up my current network storage build, I chose CoreOS (later Flatcar Container Linux) because it was an auto-updating base system with a declarative cloud-init config.

Ways of installing NixOS

Graphical Installer: Only for Desktops

The NixOS manual’s “Installation” section describes a graphical installer (“for desktop users”, based on the Calamares system installer and added in 2022) and a manual installer.

With the graphical installer, it’s easy to install NixOS to disk: just confirm the defaults often enough and you’ll end up with a working system. But there are some downsides:

You need to manually enable SSH after the installation — locally, not via the network.
The graphical installer generates an initial NixOS configuration for you, but there is no way to inject your own initial NixOS configuration.

The graphical installer is clearly not meant for remote installation or automated installation.

Manual Installation

The manual installer on the other hand is too manual for my taste: expand “Example 2” and “Example 3” in the NixOS manual’s Installation summary section to get an impression. To be clear, the steps are very doable, but I don’t want to install a system this way in a hurry. For one, manual procedures are prone to mistakes under stress. And also, copy & pasting commands interactively is literally the opposite of writing declarative configuration files.

Network Installation: nixos-anywhere

Ideally, I would want to perform most of the installation from the comfort of my own PC, meaning the installer must be usable over the network. Also, I want the machine to come up with a working initial NixOS configuration immediately after installation (no manual steps!).

Luckily, there is a (community-provided) solution: nixos-anywhere. You take care of booting a NixOS installer, then run a single command and nixos-anywhere will SSH into that installer, partition your disk(s) and install NixOS to disk. Notably, nixos-anywhere is configured declaratively, so you can repeat this step any time.

(I know that nixos-anywhere can even SSH into arbitrary systems and kexec-reboot them into a NixOS installer, which is certainly a cool party trick, but I like the approach of explicitly booting an installer better as it seems less risky and more generally applicable/repeatable to me.)

Setup: Installing Nix

I want to use NixOS for one of my machines, but not (currently) on my main desktop PC.

Hence, I installed only the nix tool (for building, even without running NixOS) on Arch Linux:

% sudo pacman -S nix
% sudo groupadd -r nixbld
% for n in $(seq 1 24); do sudo useradd -c "Nix build user $n" \
    -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(which nologin)" \
    nixbld$n; done
% sudo systemctl enable --now nix-daemon.socket

Now, running nix-shell -p hello should drop you in a new shell in which the GNU hello package is installed:

% export NIX_PATH=nixpkgs=channel:nixos-25.05
% nix-shell -p hello
hello

[nix-shell:/tmp]$ hello
Hello, world!

By the way, the Nix page on the Arch Linux wiki explains how to use nix to install packages, but that’s not what I am interested in: I only want to remotely manage NixOS systems.

Building your own installer

Previously, I said “you take care of booting a NixOS installer”, and that’s easy enough: write the ISO image to a USB stick and boot your machine from it (or select the ISO and boot your VM).

But before we can log in remotely via SSH, we need to manually set a password. I also need to SSH with the TERM=xterm environment variable because the termcap file of rxvt-unicode (my preferred terminal) is not included in the default NixOS installer environment. Similarly, my configured locales do not work and my preferred shell (Zsh) is not available.

Wouldn’t it be much nicer if the installer was pre-configured with a convenient environment?

With other Linux distributions, like Debian, Fedora or Arch Linux, I wouldn’t attempt to re-build an official installer ISO image. I’m sure their processes and tooling work well, but I am also sure it’s one extra thing I would need to learn, debug and maintain.

But building a NixOS installer is very similar to configuring a regular NixOS system: same configuration, same build tool. The procedure is documented in the official NixOS wiki.

I copied the customizations I would typically put into configuration.nix, imported the installation-cd-minimal.nix module from nixpkgs and put the result in the iso.nix file:

{ config, pkgs, ... }:

{
  imports = [
    
    
  ];

  i18n.supportedLocales = [
    "en_DK.UTF-8/UTF-8"
    "de_DE.UTF-8/UTF-8"
    "de_CH.UTF-8/UTF-8"
    "en_US.UTF-8/UTF-8"
  ];
  i18n.defaultLocale = "en_US.UTF-8";

  security.sudo.wheelNeedsPassword = false;
  users.users.michael = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5secret"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5key"
    ];

    isNormalUser = true;
    description = "Michael Stapelberg";
    extraGroups = [ "wheel" ];
    initialPassword = "SGZ3odMZIesxTuh2Y2pUaJA";  # random for this post
    shell = pkgs.zsh;
    packages = with pkgs; [];
  };

  environment.systemPackages = with pkgs; [
    git  # for checking out github.com/stapelberg/configfiles
    rsync
    zsh
    vim
    emacs
    wget
    curl
    rxvt-unicode  # for terminfo
    lshw
  ];

  programs.zsh.enable = true;
  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "25.05"; # Did you read the comment?
}

To build the ISO image, I set the NIX_PATH environment variable to point nix-build(1) to the iso.nix file and to select the upstream channel for NixOS 25.05:

% export NIX_PATH=nixos-config=$PWD/iso.nix:nixpkgs=channel:nixos-25.05
% nix-build '' -A config.system.build.isoImage

After about 1.5 minutes on my 2025 high-end Linux PC, the installer ISO can be found in result/iso/nixos-minimal-25.05.802216.55d1f923c480-x86_64-linux.iso (1.46 GB in size in my case).

Enabling Nix Flakes

Unfortunately, the nix project has not yet managed to enable the “experimental” new command-line interface (CLI) by default yet, despite 5+ years of being available, so we need to create a config file and enable the modern nix-command interface:

% mkdir -p ~/.config/nix
% echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

How can you tell old from new? The old commands are hyphenated (nix-build), the new ones are separated by a blank space (nix build).

You’ll notice I also enabled Nix flakes, which I use so that my nix builds are hermetic and pinned to a certain revision of nixpkgs and any other nix modules I want to include in my build. I like to compare flakes to version lock file in other programming environments: the idea is that building the system in 5 months will yield the same result as it does today.

To verify that flakes work, run nix shell (not nix-shell):

% nix shell nixpkgs#hello
/tmp 2 % hello
Hello, world!

(Re-)Installation Steps

For reference, here is the configuration I use to create a new VM for NixOS in Proxmox. The most important setting is bios=ovmf (= UEFI boot, which is not the default), so that I can use the same boot loader configuration on physical machines as in VMs:

Before we can boot our (unsigned) installer, we need to enter the UEFI setup and disable Secure Boot. Note that Proxmox enables Secure Boot by default, for example.

Then, boot the custom installer ISO on the target system, and ensure ssh michael@nixos.lan works without prompting for a password.

Declare a flake.nix with the following content:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-25.05";

    disko.url = "github:nix-community/disko";
    # Use the same version as nixpkgs
    disko.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs =
    {
      nixpkgs,
      disko,
      ...
    }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = false;
      };
    in
    {
      nixosConfigurations.zammadn = nixpkgs.lib.nixosSystem {
        inherit system;
        inherit pkgs;
        modules = [
          disko.nixosModules.disko
          ./configuration.nix
        ];
      };
      formatter.${system} = pkgs.nixfmt-tree;
    };
}

Declare your disk config in disk-config.nix:

disk-config.nix

Michael Stapelbergs Website: posts tagged nix

Coding Agent VMs on NixOS with microvm.nix

See also

Setting up microvm.nix

Step 1: network prep

Step 2: flake.nix

Step 3: microvm.nix

Step 4: microvm-base.nix

NixCon 2025 Trip Report 🐝

Friday

“What if GitHub Actions were local-first and built using Nix?”

“Rewriting the Hydra Queue Runner in Rust”

“You can’t spell “devshell” without “hell””

“The Nix Binary Cache and AWS”

“Nix-based development environments at Shopify (reprise)”

“My first Nix Aha!: A Newcomer’s Perspective”

Lightning Talks

Saturday

“The bikes have been shed: The official Nix formatter”

“Mastering NixOS Integration Tests: Advanced Techniques for Fast and Robust Multi-VM Tests”

“When Not to Nix: Working with External Config and SOPS Nix”

Lightning Talks

Sunday

Conclusion

Secret Management on NixOS with sops-nix

What is Secret Management?

sops-nix setup

Step 1. Preparation

Step 2. Obtain an age identity from your personal SSH key

Step 3. Obtain an age recipient for the remote machine

Step 4. Configure sops for your git repository

Step 5. Manage some secrets with sops

Step 6. Configure sops in NixOS

Usage Examples

Usage Example: command-line flags (ExecStart wrapper)

Usage Example: environment variable files

Usage Example: systemd credentials

Usage Example: samba users/passwords

Conclusion

Development shells with Nix: four quick examples

For comparison: The Debian Way

Setup: Nix-on-Debian (or Nix-on-Arch, or…)

Step 1: Install Nix

Step 2: Enable Flakes

Step 3: Set a Nix path

Example 1: Interactive one-offs: nix-shell

Example 2: nix-shell config file: shell.nix

Example 3: Hermetic, pinned devShells: Nix Flakes

Example 4: Making the Flake system-independent

Tip: Keeping packages around

Conclusion

Migrating my NAS from CoreOS/Flatcar Linux to NixOS

Context/History

My NAS Software Requirements

Why migrate from CoreOS/Flatcar to NixOS?

R1. cloud-init is deprecated

R2. Container Bitrot

R3. Dependency on a central service

R4. Could not try Immich on Flatcar

Reason Summary

Prototyping in a VM

Migrating

M1. Install NixOS

M2. Set up remote disk unlock

M3. Set up Samba for access

M4. Set up SSH/rsync for backups

Nice-to-haves

N1. Prometheus Node Exporter

N2. Reliable mounting

N3. nginx-healthz

N4. NixOS Jellyfin

N5. NixOS samba

N6. NixOS rrsync

N7. sync.pl script

N8. Sharing configs

N9. Trying immich!

Conclusion

How I like to install NixOS (declaratively)

Introduction: Declarative?

Ways of installing NixOS

Step 2: `flake.nix`

Step 3: `microvm.nix`

Step 4: `microvm-base.nix`