One flake to rule them all

Flakes are undoubtedly here to stay, and I’ve made my opinion on them well-known: they are “meh”. Despite their ever-presence, adding a flake input continues to be a small annoyance that never goes away. It’s touted as a positive feature that flakes are federated but the reality is that I want the simplicity of a centralized flake. That was the beauty and power of nixpkgs. The process is: you want disko, so you add a url, then a follows so it stops dragging in its own nixpkgs, and then you do it again for the next one. This has become a meme in the Nix community about how every flake drags in its own flake-utils. I refuse to accept this user experience. So I wondered: could one flake carry every other flake, and you just reach in for whatever you need? 🤯 §Omniflake omniflake is thousands of Nix flakes behind one flake input. inputs.omniflake.url = "github:fzakaria/omniflake"; inputs.omniflake.inputs.nixpkgs.follows = "nixpkgs"; Once you have omniflake, you can use it like any other input. A package, in a shell or on a system: environment.systemPackages = [ omniflake.flakes.nh.packages.${system}.default ]; An overlay: nixpkgs.overlays = [ omniflake.flakes.rust-overlay.overlays.default ]; A NixOS module: imports = [ omniflake.flakes.disko.nixosModules.disko ]; Or nothing in a flake at all, straight from the command line: $ nix run 'github:fzakaria/omniflake#flakes.nh.packages.x86_64-linux.default' -- --version It has an accessible website that you can visit https://omniflake.com/ to see the list of flakes it carries, and some helpful documentation. Once added, you have access to nearly twelve thousand flakes, as of this writing, that you can access as needed lazily. You only pay for what you use. This sounds absurd but it works. A flake with thousands of inputs should be unusable but thanks to the laziness of the Nix language and the flake lock mechanism it works perfectly. How is it possible to contain nearly every flake within this one?

Here is a simple flake that uses follows to unify nixpkgs: { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; disko.url = "github:nix-community/disko"; disko.inputs.nixpkgs.follows = "nixpkgs"; agenix.url = "github:ryantm/agenix"; agenix.inputs.home-manager.inputs.nixpkgs.follows = "nixpkgs"; }; } follows you your flake disko disko you->disko agenix agenix you->agenix npkgs nixpkgs disko->npkgs follows hm home-manager agenix->hm hm->npkgs follows Solid arrows are inputs; dashed are follows edges collapsing onto one shared nixpkgs. §Inputs are lazy The beauty of much of the craziness in Nix is that it is a lazy language. An input no output touches is never fetched. You can prove this destructively: lock a flake with two inputs, then corrupt one entry in flake.lock so it cannot possibly resolve. $ sed -i 's/2810303efc.../0000000000000000000000000000000000000000/' flake.lock $ nix eval .#justB [ "aarch64-darwin" "aarch64-linux" "x86_64-darwin" "x86_64-linux" ] That evaluated fine against a lock containing a revision that does not exist. Only forcing the poisoned input complains: $ nix eval .#useA error: unable to download '.../0000000000000000000000000000000000000000.tar.gz': HTTP error 404 When you add a flake as an input, its entire transitive graph is copied into your lock as metadata, nothing is fetched and nothing is evaluated. $ time nix flake lock • Added input 'mega' • Added input 'mega/a' <- the poisoned one • Added input 'mega/a/nixpkgs' real 0m0.084s §False start My first attempt at creating a massive single flake was to add every flake as an input directly into flake.nix and lock it. { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; agenix.url = "github:ryantm/agenix"; agenix.inputs.nixpkgs.follows = "nixpkgs"; disko.url = "github:nix-community/disko"; disko.inputs.nixpkgs.follows = "nixpkgs"; home-manager.url = "github:nix-community/home-manager"; home-manager.inputs.nixpkgs.follows = "nixpkgs"; # … 11,000 more … }; } Surprisingly despite the laziness of Nix, evaluating an output that touches nothing got very slow as inputs grew. The cost was not in evaluation. In order to create the lock file though the first time, Nix had to evaluate every input. When Nix creates the lock file, every node needs a unique name, and a name collision is resolved by appending _2, _3 and so forth. The code would start the search at _2 each time, and if there were 1,000 collisions, it would try _2, _3, … _1000 A megaflake guarantees collisions: every flake brings its own systems input, so 4,000 inputs produced 3,999 nodes named systems_2 through systems_4000, about 8 million string formats per evaluation. This turned out to be quadratic time complexity in the number of inputs, and it was the reason for the slowdown. The fix was relatively simple: remember the highest suffix used per name and resume from there. I submitted NixOS/nix#16387, and the lock files it writes are byte-identical to before. 1980-01-01T00:00:00+00:00 image/svg+xml Matplotlib v3.10.5, https://matplotlib.org/ No one had been writing absurdly large flakes, so this quadratic cost was invisible. The fix was relatively simple and the speedup is dramatic: ~21x faster for 4,000 inputs. Despite this fix, this turned out to be a false start. In order to finish creating the flake.lock file, Nix has to evaluate every input. Nix locks one input at a time, fetching each tree to read its flake.nix, and one input that cannot be locked aborts the run.11I tried assembling the lock file outside Nix from the flake.lock files directly for each input but it kept disagreeing with nix flake lock and would cause Nix to redo the entire lock. So I went to the source to find out what a consumer does with an inherited lock. The answer is less than I assumed. When you add a flake as an input, Nix checks that flake’s direct inputs against its flake.nix, and copies everything deeper into your lock unread.