From 8c0ee05d546400f76b77317f55fa1232c19ec800 Mon Sep 17 00:00:00 2001 From: Franck Cuny Date: Tue, 10 Jan 2023 09:04:27 -0800 Subject: feat(content): Workaround md raid boot issue in NixOS 22.11 --- content/blog/nix-raid-systemd-boot.md | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 content/blog/nix-raid-systemd-boot.md (limited to 'content/blog') diff --git a/content/blog/nix-raid-systemd-boot.md b/content/blog/nix-raid-systemd-boot.md new file mode 100644 index 0000000..b31d29c --- /dev/null +++ b/content/blog/nix-raid-systemd-boot.md @@ -0,0 +1,48 @@ +--- +title: Workaround md raid boot issue in NixOS 22.11 +date: 2023-01-10 +tags: +- nixos +--- +For about a year now I've been running [NixOS](https://nixos.org/ "NixOS") on my personal machines. Yesterday I decided to go ahead and upgrade my NAS from NixOS 22.05 to [22.11](https://nixos.org/blog/announcements.html#nixos-22.11). On that machine, all the disks are encrypted, and there are two RAID0 devices. To unlock the drives, I log into the [SSH daemon running in `initrd`](https://nixos.wiki/wiki/Remote_LUKS_Unlocking), where I can type my passphrase. This time however, instead of a prompt to unlock the disk, I see the following message: +``` +waiting for device /dev/disk/by-uuid/66c58a92-45fe-4b03-9be0-214ff67c177c to appear... +``` +followed by a timeout and then I'm asked if I want to reboot the machine. I do reboot the machine, and same thing happens. + +Now, and this is something really great about NixOS, I can boot to the previous generation (on 22.05), and this time I'm prompted for my password, the disks are unlocked, and I can log into my machine. This eliminates the possibility of a hardware failure! I also have a way to get a working machine to do more build if needed. Knowing that I can easily switch from a broken generation to a working one gives me more confidence in making changes to my system. + +I then reboot again in the broken build, and drop into a `busybox` shell. I look to see what `blkid` reports, and I confirm that my disks are all present and they have a **UUID** set. Next I check what's listed under `/dev/disk/by-uuid` and, surprise, the disks are not there. They are however under `/dev/disk`. Now, looking at `/nix/store` I only see a few things, and one of them is a script named `stage-1-init.sh`. I read quickly the script, checked it does, and confirmed that it was blocking on the disks. I looked at what was reported by `udevadm info ` and I could see that the `DEVLINKS` was missing the path for `by-uuid`. + +My laptop has a similar setup, but without RAID devices. I had already updated to 22.11, and had rebooted the laptop without issues. To be sure, I ran another update and rebooted, and I was able to unlock the drive and log into the machine without problem. + +From here I have enough information to start searching for an issue similar to this. I got pretty lucky and two issues I found were: +- [Since systemd-251.3 mdadm doesn't start at boot time #196800 ](https://github.com/nixoS/nixpkgs/issues/196800) +- [Won't boot when root on raid0 with boot.initrd.systemd=true #199551 ](https://github.com/nixoS/nixpkgs/issues/199551) + +The proposed solution was easy: +```diff +@@ -43,7 +43,7 @@ + }; + + boot.initrd.luks.devices."raid-fast".device = +- "/dev/disk/by-uuid/66c58a92-45fe-4b03-9be0-214ff67c177c"; ++ "/dev/disk/by-id/md-name-nixos:fast"; + + fileSystems."/data/slow" = { + device = "/dev/disk/by-uuid/0f16db51-0ee7-48d8-9e48-653b85ecbf0a"; +@@ -51,7 +51,7 @@ + }; + + boot.initrd.luks.devices."raid-slow".device = +- "/dev/disk/by-uuid/d8b21267-d457-4522-91d9-5481b44dd0a5"; ++ "/dev/disk/by-id/md-name-nixos:slow"; +``` + +I rebuild, rebooted, and success, I was able to get access to the machine. + +## Takeaways + +I now have a mitigation to the problem, however I still don't have a root cause. Since it's only the `by-uuid` path that is missing, and this is managed by `udev`, I'm guessing that some rules for `udev` have changed, but so far I can't find anything about that. + +It's really great to be able to easily switch back to a previous generation of my system, so I can debug and experiment different solutions. If this had happen with another distribution, getting out of this mess would have been more tedious. -- cgit 1.4.1