Setting Up Nix Dev Environment for Middleman and Ruby

Wanna know why I've stopped working on Advent of Code on Day 8? This problem right here! 🤦

So when I initially switched to Nix for dotfiles and setup NixOS I was able to set up the blog, but I wasn't reaaaaally using Nix. I used Nix to set up my trusty asdf-vm and used that to handle ruby and bundler to handle gem depedencies.

While working on more complicated projects I discovered this approach doesn't work well. It really hangs up if the ruby gem wants to build a native extension… which a lot of things in rails does.1

The More Correct Answer™️ is to let Nix handle all dependencies, including gems and NPM packages.

I initially went down the path of looking at cachix/devenv and then jetpack-it/devbox. Both are very nice, but I couldn't make much headway, I suspect because I didn't really understand enough about Nix yet.2

As it was, I fought for weeks chasing different Nix errors. Sometimes I'd get it working on one device, but not another. My "lab journal" note in drafts and commit logs are filled with experiments, guesses and frustrations.

Here's where I landed:

I stopped using either devenv or devbox and wrote a simple flake based on the one from the documentation.

{
  description = "evantravers.com (middleman, ruby, node)";

  inputs = {
    nixpkgs.url = "nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        rubyEnv = pkgs.bundlerEnv {
          # The full app environment with dependencies
          name = "middleman-env";
          inherit (pkgs) ruby;
          gemdir = ./.; # Points to Gemfile.lock and gemset.nix
        };

        buildPackages = [
          pkgs.bundix
          rubyEnv
          rubyEnv.wrappedRuby
          pkgs.nodejs
        ];
      in with pkgs;
      {
        devShells.default =
          mkShell {
            env = {
              "NODE_OPTIONS" = "--openssl-legacy-provider";
            };
            buildInputs = buildPackages ++ [
              pkgs.solargraph
            ];
          };

      packages.default =
        stdenv.mkDerivation {
          name = "build the website";
          src = ./.;
          buildInputs = buildPackages;
          buildPhase = "${rubyEnv}/bin/middleman build";
      };
    }
  );
}

The way to get ruby dependencies into nix is as follows:

  1. Drop into a nix shell that has a few packages: nix-shell -p bundix ruby.
  2. Now we have access to bundler, so run bundle lock to generate a Gemfile.lock.
  3. That gemfile.lock describes the location and identity of dependencies: bundix -l translates that into a nix expression in gemset.nix.
  4. In the flake you need to import a ruby with that gemset.nix expression as part of the environment (the bundlerEnv above.)

Seems straightforward right? Well…

My flake can't be used to create a shell with the dependencies needed to debug until it compiles cleanly, so you need to keep using nix-shell. When it works, it works. But until then…

So what was going wrong for so long? Some of my mistakes:

Force ruby platform

For a long time I was wrestling on and off with various ruby gems that were building native extensions or had platform specific versions.

I didn't solve this problem, I just sidestepped it.

I ran bundle config set --local force_ruby_platform true, re-ran bundle lock and removed any references to individual platforms. For this simple blog, that worked fine. It won't work for a more complicated situation, so I'll learn more about that later.

You have to include the gemset.nix

For a long time I was generating the gemset.nix with bundix, but then not including it in the flake. 🤦 After I had used the let portion to define the bundlerEnv, I needed to include it in the buildPackages portion of the mkShell. I was defining all the gems… but not using them.

Stop using bundler

I kept running bundle exec like an idiot.

For years I have dodged so many ruby bullets by simply running bundle exec in front of every executable call and it's become muscle memory. By doing this I ensure that the local bundle calls the gems I specified in this project's gemfile, not some global copy floating somewhere else.

Except here… I don't want to use the local gems I pulled with bundle install… in fact I don't even want to install them using bundler. All bundler needs to do is describe the dependency graph to be translated by bundix into a nix expression. That's it.

For a bit bundle exec middleman build was indeed working… because the gems installed into ./vendor were sufficient to run that code. As I dug into some weird issues with bundle exec middleman server I saw that the stack trace included gems in ./vendor/cache and in /nix/store… whoopsie.

Just match the mismatched sha256

For a long time I was fighting an error in my flake where it was getting a sha different than what it expected. I don't know why it happened: it might be that my particular version of bundler wasn't handling platform-specific gems correctly, or that bundix wasn't interpreting the file correctly.

Complicated causes don't require complicated solutions. After reading technogothic's post, I took the SHA that nix said it was expecting and put it into the gemset.nix. Problem solved.

You can't write to read-only

After all this… I could run middleman server and everything started! I happily opened localhost:4567 in my browser aaaaand nothing. If I interrupted the server I got a cryptic stacktrace about files, basenames, recursion… and it was slightly different every time?

I kept googling traces, messing with things, and remembered I'd seen this before. I ran middleman build and it worked just fine.

Do I learn from my mistakes? Slowly. This is the same problem that I faced when trying to use Lazy.nvim. Nix was mounting my website's files and dependencies in the immutable nix store, and something in middleman's livereload server really hates that.

For now, I can run middleman server --watcher-disable and move on with my life.

…

There's been a string of long-running late-night texts between me and friends over the past month as I fought to bring my blog into Nix.

Me

(you probably know all this crap… but it feels like a breakthru to me)

Friend

yea

You've never dated a crazy girl.. but that's what it feels like

the highs are high and the lows are low

🤔

He's not totally wrong. Another friend, @qmx, warned me to stay away from Nix… at the moment I'm persevering through the pain. At the moment I haven't found a solution to the middleman live-reload watcher working again…3 but if I freeze the flake.lock I can guarantee that my blog will at least run for a long time even though middleman's future is in question, and that's a good feeling.4

I'm going to try to stand up a more complicated rails app with databases, caches, and a separate microservice… we'll see how I feel about it after that.

If all else fails, I can fall back to letting nix stand up services, define asdf-vm, and keep rolling like I always have, but I want the reproducibility… and longevity that nix promises. I started this on a WSL NixOS, worked on it on an old intel Mac, and am finishing the spellchecking on a M1 ARM Mac... and for each platform all I did was cd into the folder, direnv triggered the nix evaluation, and I could start work.

It's a bright promise, we'll see if it holds up to be a real future.

References


  1. This was the same problem that I encountered using Lazy.nvim to try and install LSP grammars. Nix is evaluating everything in its read only store… so temp build files break its functional perfection. 

  2. If I go back to one, it'll probably be devenv. Devbox's decision to abstract away nix expressions and only use json means I'm not sure how to use Bundix with it. I guess I'd have to write a custom package and use that as an input in the Devbox's json? Unsure. Devbox is available as a package in home-manager, devenv has nice caching… problem for the next project that requires services and databases.  

  3. I could probably use direnv to reload the server every time change a file, but that feels heavy. I just need to dive into middleman server's documentation… I could probably have it write a cache folder I control. 

  4. Gives me the confidence to extend my custom admonition markup to ape Xe's cute conversations


đź”–
Changelog
  • 2024-01-04 19:27:08 -0600
    fix

  • 2024-01-04 15:19:31 -0600
    Name the links

  • 2024-01-04 15:12:45 -0600
    Fix width issue

  • 2024-01-04 14:56:09 -0600
    Tag and release

  • 2024-01-04 14:04:47 -0600
    writing

  • 2024-01-04 13:14:29 -0600
    working in the doctor's office

  • 2024-01-04 12:35:10 -0600
    Work on the blog post

  • 2024-01-04 08:01:45 -0600
    Writing the blog post

  • 2024-01-04 07:44:44 -0600
    Draft post