Migrating off NixOS
About a year ago, I adopted NixOS as the underlying OS for my daily work machine. My setup is a bit varied, but I actually use an M2 Mac Studio as my "visual" driver and have NixOS installed on a beefy tower (32c/128GB RAM), which I work from using SSH and VSCode Remote.
A Quick Recap
I first chose to adopt NixOS because many of my day job colleagues used it as their daily driver. Nix holds a prominent position in our company culture, and some teams use it to do everything (i.e., cover the full SDLC). When I began onboarding into the company, embracing Nix and seeing what it offered made sense.
My first few months with Nix were mostly positive. Nix is known to be a complex language to pick up, and my experience mostly matches that. It didn't help that Nix was my first functional and lazily evaluated language. I appreciate Nix for introducing me to both, as I've since used that foundational knowledge to grow in other areas (i.e., CUE).
Things I Like
By far, my favorite feature of Nix is the portable development shells. As an SRE, I work across dozens of repositories, and each one tends to use different development tools. The combination of Nix development shells with something like direnv makes working across these repositories much simpler. For example, when I move into a deployment-focused repository, I automatically get tools like Terraform and the AWS CLI loaded into my $PATH. More importantly, these tools do not have to exist anywhere in the global scope of my system.
Second to the development shells is a popular Nix project for managing my home environment: home-manager. This was a game changer because I never found an enjoyable way to manage my dotfiles. Not only does home-manager offer a plethora of modules for configuring standard tools, but it also provides an easy way to make packages available to my local user, much like a Nix development shell. Managing my shell configuration, local packages, and various configurations from a single place makes for a very pleasant UX.
Things That Bother Me
All this being said, over the last year, I've also developed a laundry list of things that bother me about Nix. I have no desire to document all of them here, but I'll leave a high-level list.
First, working with the Nix language and its associated tooling is an exercise in patience. Not only is the documentation abysmal in most cases, but forming reasonable assumptions about how something should work is often challenging. Over time, these factors compounded to reduce my agility in accomplishing daily tasks with Nix. Indeed, after a few months, I dreaded any time a Nix stack trace appeared. Depending on the complexity of the underlying code, the stack trace was often hundreds of lines long and, in many cases, completely hid the underlying problem. When I'm in the middle of a task, especially a time-sensitive one, stopping to troubleshoot Nix begins to get old quickly.
Second, Nix is very opinionated about how to build software. This is more or less a design constraint because it eschews non-reproducible builds. Philosophically, this is a nice thought. However, I've found the reality of these decisions hard to stomach. For example, anyone who has tried to build a NodeJS-based project with Nix can attest to the pain of the endeavor. I am not exaggerating when I say that my developers would break the build multiple times per week. Over three months, I switched the underlying Nix-based project that was building the code numerous times, and I never once found one that was even remotely stable. The issue wasn't isolated to a single language: in another instance, I had a repository that contained multiple Rust crates and used a root-level Cargo.toml to configure everything. It turns out that, at the time, none of the Nix-based Rust builders supported this architecture, and I had to spend over a week hacking together a solution. Again, this all adds up to wasted cycles and delayed me from getting to the meaningful work I had ahead of me.
Third, which is similar to the last point, NixOS itself is also very opinionated about how an operating system should be structured. Practically speaking, the FHS format isn't compatible with Nix's assumptions. Instead, everything lives in the Nix store, and NixOS resorts to all sorts of complex wrappers to emulate what the average piece of software expects to see. If you happen to stay within the software supported by nixpkgs, you're more or less okay. However, if you inadvertently drop a binary onto a NixOS machine that wasn't compiled within nixpkgs, you're in for a nasty surprise. This is because a large portion of the software is dynamically linked with the expectation a system will respect the typical FHS locations of shared dependencies. Whether this is ideal or not is beside the point because this is how most software is distributed. So, for example, if the VSCode Remote server decides to download a specific version of the NodeJS runtime, it will fail to run on a NixOS machine. There are very complex hacks to avoid this, but the problem is that they tend to break anytime Microsoft makes a change. In my case, I've had my remote development broken several times and often been left with no solution for days. This is entirely unacceptable for my line of work.
Suffice it to say, I could keep going, but the general sentiment is this: Nix and NixOS are too problematic for daily driving. I spend far too much time troubleshooting a poorly designed language with stringent opinions about how the world should operate. Using it in any mission-critical or production environment is, in my opinion, more trouble than it's worth.
Moving On
So, what's next? I'm currently heavily looking into Bluefin Linux. I have no qualms with reproducible systems, and in fact, it's something I'm actively looking for in my daily driver. While OSTree has its issues, it's less opinionated than Nix and at least stays within the bounds of a typical Linux system while still offering safety around permanently breaking a system.
A major benefit of Bluefin is it espouses Nix for all of the reasons I love: as a manager of my home environment and for setting up ad-hoc development shells. In this way, I can get everything I love about Nix, without adopting everything I dislike about it.
One other thing I didn't cover in the previous section is the disjointed Nix community, which consistently competes with itself to the detriment of others. Bluefin is ultimately based on Fedora Silverblue, which is maintained by Red Hat. This gives me a lot of confidence in the long-term stability of the project. Combine this with a group of CNCF advocates supporting the Bluefin-specific items, and I think the community that will evolve around this OS will be much more in line with how I use Linux daily.